doppler issue which fails to emit logs with syslog protocol on CFv212


Masumi Ito
 

Hi, I am investigating the doppler issue on CFv212 that sending logs to the
external logging service with syslog protocol fails.

As my understanding, the following messages are supposed to be recorded in
the "doppler.stdout.log" if the doppler got the syslog drain url from etcd
successfully.
However it is actually missing.

- Missing log messages which is expected to be shown
{"timestamp":xxxxxx,"process_id":xxxx,"source":"doppler","log_level":"info","message":"Syslog
Sink syslog://xx.xx.xx.xx:xxxx:
Running.","data":null,"file":"/var/vcap/data/compile/doppler/loggregator/src/doppler/sinks/syslog/syslog_sink.go","line":56,"method":"doppler/sinks/syslog.(*SyslogSink).Run"}
{"timestamp":xxxxxx,"process_id":xxxx,"source":"doppler","log_level":"info","message":"Syslog
Sink syslog://xxxxxx:xxxxxx: successfully
connected.","data":null,"file":"/var/vcap/data/compile/doppler/loggregator/src/doppler/sinks/syslog/syslog_sink.go","line":112,"method":"doppler/sinks/syslog.(*SyslogSink).Run"}

Instead, there are a lot of etcd error events found as follows.

{"timestamp":xxxxxxxxx,"process_id":xxx,"source":"doppler","log_level":"error","message":"AppStoreWatcher:
Got error while waiting for ETCD events: store request timed
out","data":null,"file":"/var/vcap/data/compile/doppler/loggregator/src/github.com/cloudfoundry/loggregatorlib/store/app_service_store_watcher.go","line":79,"method":"github.com/cloudfoundry/loggregatorlib/store.(*AppServiceStoreWatcher).Run"}

I have two question about this.

=============
Q1) Does anyone know what this event indicates and how it affects the CF
environment?
In the normal envrionment, is this event still triggered (In other
words, can we ignore this error event messages?)

Q2) If the etcd got some trouble at the moment, which cf component is also
made an influence on?

I guess at the least the following cf components could be affected. Do we
have anyting else?
- router : to support routing api
- hm9000 : to support health check
- doppler: to get syslog drain urls from etcd
- syslog-binder to get syslog drain urls from cc and then store them to
etcd
- trafficcontroller, metron agents : to find healthy dopplers to access
to)
=============

Note that I am also doubting the following errors in the
"syslog_drain_binder.stdout.log".
This message indicates that syslog_drain_binder failed to get syslog drain
urls from cc.
{"timestamp":xxxx,"process_id":xxxx,"source":"syslog_drain_binder","log_level":"error","message":"Error
when polling cloud controller: Remote server error:
Unauthorized","data":null,"file":"/var/vcap/data/compile/syslog_drain_binder/loggregator/src/syslog_drain_binder/main.go","line":68,"method":"main.main"}

Therefore I have not yet concluded the etcd mainly caused the issue, however
need to understand the exact meaning of the error event message above as
well as any impact on the cf envrionment if there is something wrong with
the behaviour of etcd.

Regards,
Masumi



--
View this message in context: http://cf-dev.70369.x6.nabble.com/doppler-issue-which-fails-to-emit-logs-with-syslog-protocol-on-CFv212-tp2418.html
Sent from the CF Dev mailing list archive at Nabble.com.

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.