Date
1 - 9 of 9
Syslog Drain to Logstash Problems
Steve Wall <steve.wall@...>
Hello,
We are having problems draining log messages to Logstash. The drain is setup as a user provided service. cf cups logstash-drain -l syslog://xx.xx.xx.xx:5000 And then bound to the service. cf bind-service myapp logstash-drain But no log messages are coming through to Logstash. Or more specifically, we are using ELK and the messages aren't seen through Kibana. We were able to log into the DEA and using netcat (nc), messages were successfully submitted to the ELK stack. nc -w0 -u xx.xx.xx.xx 5000 <<< "logging from remote" Any suggestions on how to debug this further? -Steve |
|
John Tuley <jtuley@...>
Steve,
Until recently (cf-release v198), binding a syslog service required restarting the app. If you're post-v198, it *should* Just Work. However, one of the things that could be in your way is network security. In order to forward logs to your drain, your loggregator servers must be able to access that server. This is the most common cause we see of systems failing to forward to syslog drains. Please let us know if you have more questions. – John Tuley On Wed, Jun 3, 2015 at 12:37 PM, Steve Wall < steve.wall(a)primetimesoftware.com> wrote: Hello, |
|
Josh Ghiloni
We’ll check that, thanks!
toggle quoted message
Show quoted text
Josh Ghiloni Senior Consultant 303.932.2202 o | 303.590.5427 m | 303.565.2794 f jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com> ECS Team Technology Solutions Delivered ECSTeam.com<http://ECSTeam.com> On Jun 3, 2015, at 15:41, John Tuley <jtuley(a)pivotal.io<mailto:jtuley(a)pivotal.io>> wrote:
Steve, Until recently (cf-release v198), binding a syslog service required restarting the app. If you're post-v198, it should Just Work. However, one of the things that could be in your way is network security. In order to forward logs to your drain, your loggregator servers must be able to access that server. This is the most common cause we see of systems failing to forward to syslog drains. Please let us know if you have more questions. – John Tuley On Wed, Jun 3, 2015 at 12:37 PM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote: Hello, We are having problems draining log messages to Logstash. The drain is setup as a user provided service. cf cups logstash-drain -l syslog://xx.xx.xx.xx:5000 And then bound to the service. cf bind-service myapp logstash-drain But no log messages are coming through to Logstash. Or more specifically, we are using ELK and the messages aren't seen through Kibana. We were able to log into the DEA and using netcat (nc), messages were successfully submitted to the ELK stack. nc -w0 -u xx.xx.xx.xx 5000 <<< "logging from remote" Any suggestions on how to debug this further? -Steve _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev |
|
Steve Wall <steve.wall@...>
I was able submit a log entry from the loggregator VM using -
toggle quoted message
Show quoted text
nc -w0 10.xx.xx.xx 5000 <<< "logging from loggregator" and to test UDP nc -u -w0 10.xx.xx.xx 5000 <<< "logging from loggregator" Which leads me to believe the networking is working properly. Any other thoughts? Thanks! Steve On Wed, Jun 3, 2015 at 6:14 PM, Josh Ghiloni <jghiloni(a)ecsteam.com> wrote:
We’ll check that, thanks! |
|
MJ
We had the same problem due to missing templates in our manifest.
toggle quoted message
Show quoted text
We initially used the example manifest snippet shown at https://github.com/cloudfoundry/loggregator which mentions only the doppler template. Looking at https://github.com/cloudfoundry/loggregator/blob/develop/manifest-templates/cf-lamb.yml we later determined that we also needed the syslog_drain_binder and metron_agent templates for a complete loggregator deployment. -Mike On Jun 10, 2015, at 9:35 AM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote:
I was able submit a log entry from the loggregator VM using - nc -w0 10.xx.xx.xx 5000 <<< "logging from loggregator" and to test UDP nc -u -w0 10.xx.xx.xx 5000 <<< "logging from loggregator" Which leads me to believe the networking is working properly. Any other thoughts? Thanks! Steve On Wed, Jun 3, 2015 at 6:14 PM, Josh Ghiloni <jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>> wrote: We’ll check that, thanks! Josh Ghiloni Senior Consultant 303.932.2202<tel:303.932.2202> o | 303.590.5427<tel:303.590.5427> m | 303.565.2794<tel:303.565.2794> f jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com> ECS Team Technology Solutions Delivered ECSTeam.com<http://ecsteam.com/> On Jun 3, 2015, at 15:41, John Tuley <jtuley(a)pivotal.io<mailto:jtuley(a)pivotal.io>> wrote: Steve, Until recently (cf-release v198), binding a syslog service required restarting the app. If you're post-v198, it should Just Work. However, one of the things that could be in your way is network security. In order to forward logs to your drain, your loggregator servers must be able to access that server. This is the most common cause we see of systems failing to forward to syslog drains. Please let us know if you have more questions. – John Tuley On Wed, Jun 3, 2015 at 12:37 PM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote: Hello, We are having problems draining log messages to Logstash. The drain is setup as a user provided service. cf cups logstash-drain -l syslog://xx.xx.xx.xx:5000 And then bound to the service. cf bind-service myapp logstash-drain But no log messages are coming through to Logstash. Or more specifically, we are using ELK and the messages aren't seen through Kibana. We were able to log into the DEA and using netcat (nc), messages were successfully submitted to the ELK stack. nc -w0 -u xx.xx.xx.xx 5000 <<< "logging from remote" Any suggestions on how to debug this further? -Steve _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev |
|
John Tuley <jtuley@...>
I don't have an answer, so here are some diagnostic questions to help you
debug: Have you set doppler.blacklisted_syslog_ranges to something that includes your destination? (This would be bad.) Do you have syslog_drain_binder running? (It should be colocated with doppler on the loggregator VMs.) Does it seem healthy? Can you use the HTTP API to etcd to look at the keys in /loggregator/services. You should see an entry per (app ID, drain URL) pair. If those aren't present, then the syslog drain binder isn't working, and doppler won't get bindings. If you have the collector enabled, look in your metrics for one containing "DopplerServer.messageRouter.numberOfSyslogSinks". If that number seems accurate, then it's likely networking (which you have already ruled out) or something really weird in the code. – John Tuley On Wed, Jun 10, 2015 at 10:35 AM, Steve Wall < steve.wall(a)primetimesoftware.com> wrote: I was able submit a log entry from the loggregator VM using - |
|
John Tuley <jtuley@...>
Mike,
toggle quoted message
Show quoted text
Thanks for finding that. I've filed a bug <https://www.pivotaltracker.com/story/show/96801752> to get the README fixed. – John Tuley On Thu, Jun 11, 2015 at 10:22 AM, Mike Jacobi <jacobi(a)adobe.com> wrote:
We had the same problem due to missing templates in our manifest. |
|
Josh Ghiloni
It appears that we do not have the syslog_drain_binder templates as Mike mentioned — thanks guys! We’ll update that and give ‘er a go.
toggle quoted message
Show quoted text
Josh Ghiloni Senior Consultant 303.932.2202 o | 303.590.5427 m | 303.565.2794 f jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com> ECS Team Technology Solutions Delivered ECSTeam.com<http://ECSTeam.com> On Jun 11, 2015, at 11:10, John Tuley <jtuley(a)pivotal.io<mailto:jtuley(a)pivotal.io>> wrote:
Mike, Thanks for finding that. I've filed a bug<https://www.pivotaltracker.com/story/show/96801752> to get the README fixed. – John Tuley On Thu, Jun 11, 2015 at 10:22 AM, Mike Jacobi <jacobi(a)adobe.com<mailto:jacobi(a)adobe.com>> wrote: We had the same problem due to missing templates in our manifest. We initially used the example manifest snippet shown at https://github.com/cloudfoundry/loggregator which mentions only the doppler template. Looking at https://github.com/cloudfoundry/loggregator/blob/develop/manifest-templates/cf-lamb.yml we later determined that we also needed the syslog_drain_binder and metron_agent templates for a complete loggregator deployment. -Mike On Jun 10, 2015, at 9:35 AM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote: I was able submit a log entry from the loggregator VM using - nc -w0 10.xx.xx.xx 5000 <<< "logging from loggregator" and to test UDP nc -u -w0 10.xx.xx.xx 5000 <<< "logging from loggregator" Which leads me to believe the networking is working properly. Any other thoughts? Thanks! Steve On Wed, Jun 3, 2015 at 6:14 PM, Josh Ghiloni <jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>> wrote: We’ll check that, thanks! Josh Ghiloni Senior Consultant 303.932.2202<tel:303.932.2202> o | 303.590.5427<tel:303.590.5427> m | 303.565.2794<tel:303.565.2794> f jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com> ECS Team Technology Solutions Delivered ECSTeam.com<http://ecsteam.com/> On Jun 3, 2015, at 15:41, John Tuley <jtuley(a)pivotal.io<mailto:jtuley(a)pivotal.io>> wrote: Steve, Until recently (cf-release v198), binding a syslog service required restarting the app. If you're post-v198, it should Just Work. However, one of the things that could be in your way is network security. In order to forward logs to your drain, your loggregator servers must be able to access that server. This is the most common cause we see of systems failing to forward to syslog drains. Please let us know if you have more questions. – John Tuley On Wed, Jun 3, 2015 at 12:37 PM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote: Hello, We are having problems draining log messages to Logstash. The drain is setup as a user provided service. cf cups logstash-drain -l syslog://xx.xx.xx.xx:5000 And then bound to the service. cf bind-service myapp logstash-drain But no log messages are coming through to Logstash. Or more specifically, we are using ELK and the messages aren't seen through Kibana. We were able to log into the DEA and using netcat (nc), messages were successfully submitted to the ELK stack. nc -w0 -u xx.xx.xx.xx 5000 <<< "logging from remote" Any suggestions on how to debug this further? -Steve _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev |
|
Josh Ghiloni
That did solve our problems. Thanks everyone!
toggle quoted message
Show quoted text
Josh Ghiloni Senior Consultant 303.932.2202 o | 303.590.5427 m | 303.565.2794 f jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com> ECS Team Technology Solutions Delivered ECSTeam.com<http://ECSTeam.com> On Jun 12, 2015, at 09:20, Josh Ghiloni <jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>> wrote:
It appears that we do not have the syslog_drain_binder templates as Mike mentioned — thanks guys! We’ll update that and give ‘er a go. Josh Ghiloni Senior Consultant 303.932.2202 o | 303.590.5427 m | 303.565.2794 f jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com> ECS Team Technology Solutions Delivered ECSTeam.com<http://ecsteam.com/> On Jun 11, 2015, at 11:10, John Tuley <jtuley(a)pivotal.io<mailto:jtuley(a)pivotal.io>> wrote: Mike, Thanks for finding that. I've filed a bug<https://www.pivotaltracker.com/story/show/96801752> to get the README fixed. – John Tuley On Thu, Jun 11, 2015 at 10:22 AM, Mike Jacobi <jacobi(a)adobe.com<mailto:jacobi(a)adobe.com>> wrote: We had the same problem due to missing templates in our manifest. We initially used the example manifest snippet shown at https://github.com/cloudfoundry/loggregator which mentions only the doppler template. Looking at https://github.com/cloudfoundry/loggregator/blob/develop/manifest-templates/cf-lamb.yml we later determined that we also needed the syslog_drain_binder and metron_agent templates for a complete loggregator deployment. -Mike On Jun 10, 2015, at 9:35 AM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote: I was able submit a log entry from the loggregator VM using - nc -w0 10.xx.xx.xx 5000 <<< "logging from loggregator" and to test UDP nc -u -w0 10.xx.xx.xx 5000 <<< "logging from loggregator" Which leads me to believe the networking is working properly. Any other thoughts? Thanks! Steve On Wed, Jun 3, 2015 at 6:14 PM, Josh Ghiloni <jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>> wrote: We’ll check that, thanks! Josh Ghiloni Senior Consultant 303.932.2202<tel:303.932.2202> o | 303.590.5427<tel:303.590.5427> m | 303.565.2794<tel:303.565.2794> f jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com> ECS Team Technology Solutions Delivered ECSTeam.com<http://ecsteam.com/> On Jun 3, 2015, at 15:41, John Tuley <jtuley(a)pivotal.io<mailto:jtuley(a)pivotal.io>> wrote: Steve, Until recently (cf-release v198), binding a syslog service required restarting the app. If you're post-v198, it should Just Work. However, one of the things that could be in your way is network security. In order to forward logs to your drain, your loggregator servers must be able to access that server. This is the most common cause we see of systems failing to forward to syslog drains. Please let us know if you have more questions. – John Tuley On Wed, Jun 3, 2015 at 12:37 PM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote: Hello, We are having problems draining log messages to Logstash. The drain is setup as a user provided service. cf cups logstash-drain -l syslog://xx.xx.xx.xx:5000 And then bound to the service. cf bind-service myapp logstash-drain But no log messages are coming through to Logstash. Or more specifically, we are using ELK and the messages aren't seen through Kibana. We were able to log into the DEA and using netcat (nc), messages were successfully submitted to the ELK stack. nc -w0 -u xx.xx.xx.xx 5000 <<< "logging from remote" Any suggestions on how to debug this further? -Steve _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev |
|