Re: doppler issue which fails to emit logs with syslog protocol on CFv212
We setup a similar environment with CFv212 and went through a few scenarios. We took a look at the problem and we feel that there are couple of odd things.
"Error when polling cloud controller: Remote server error: Unauthorized" is caused when syslog_drain_binder gets a 401 response from CC which means your credentials could be invalid.
"AppStoreWatcher: Got error while waiting for ETCD events: store request timed out" is caused when we took ETCD down and noticed a flood of these errors in the doppler.stdout.log. In this case (ETCD being down) you should also see in syslog_drain_binder.stdout.log the following error message: "Error when staying leader: store request timed out". And yes other components like trafficcontroller and metron agents should also error out.
How many ETCDs were running at the time of these errors? Was there a deployment going on at the time? As Dies Koper pointed out (lamb slack channel), in CF 212 the username for the syslog drain endpoint in the syslog_drain_binder is hardcoded to bulk_api and only the password is configurable. So maybe there was a password mismatch? Check the timestamps of the events to see if they are related.
Warren & Nino