Date
1 - 4 of 4
Log drain for an app
|
Daniel Mikusa
Hello,
I'm looking at sending logs from my app to Logsene [1] and I'm trying to figure out if this is going to work. From their instructions it seems like there are two possibilities: syslog & https I'm not sure syslog will work as Logsene seems to either require a unique token to be included in the syslog event or to have all syslog traffic from my app come from one IP. I'm not sure that the first is possible, and the second won't work as multiple apps on CF could send logs from the same IP (, please correct me if I'm wrong on either point). That leaves me with HTTPS. According to their docs, they support the elasticsearch api [2] through which you can post events to them. It seems to expect a JSON payload, with a standard format. I see in the CF docs [3] that we support sending logs via HTTPS but it doesn't really say how the information is sent via HTTPS. Does anyone know if this will be compatible? and where I can find more information about how we send log data via HTTPS? Thanks Dan [1] - https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Logsene+documentation%3A+Home [2] - https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Index+Events+via+Elasticsearch+API [3] - http://docs.cloudfoundry.org/devguide/services/log-management.html#config |
|
|
|
John Tuley <jtuley@...>
Dan,
toggle quoted message
Show quoted text
I see three questions in your email, which I'll try to address in turn: - *"Can my application send logs with a unique token?"* – Your application can add any text it likes to the message, of course. When they're sent to the syslog drain, the messages will be embedded in a syslog-formatted line. Looking at Logsene's syslog example <https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Syslog#Syslog-Example>, it seems that they expect the syslog message to contain a JSON payload with the token as a property. If your application produces that JSON, I think it would be compatible. However, the Loggregator system does not wrap bare loglines into that format, nor can it be configured to do so without rewriting code. - *"Do multiple apps on CF send logs from the same IP address?"* – Yes. But it's worse than that: not only do multiple app streams come from the same IP address, but a single application's stream can come from multiple IP addresses. So this is probably not good from Logsene's point of view. - *"Is Loggregator's HTTPS transport compatible with the ElasticSearch API?"* – No. Loggregator makes a POST request to the HTTPS endpoint by putting a syslog-formatted line into the body of the request. It does not have support for building an ElasticSearch-compatible JSON payload around the message. It appears to me that the best shot you have of compatibility with Logsene is having your application build messages in the expected way, with JSON wrapper (if that's truly needed; my quick read of the syslog example I linked above was unclear). Keep in mind that Loggregator sends each *line* separately, so your JSON payload must be a single line to be transmitted correctly. – John Tuley On Fri, Jun 12, 2015 at 11:13 AM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:
Hello, |
|
|
|
Daniel Mikusa
Thanks John! I'll play with that and see if I can make it work.
toggle quoted message
Show quoted text
Dan On Fri, Jun 12, 2015 at 4:26 PM, John Tuley <jtuley(a)pivotal.io> wrote:
Dan, |
|
|
|
Daniel Mikusa
John,
Thanks again! I was able to get this working. Some notes below if anyone else is trying this out. I see three questions in your email, which I'll try to address in turn:1.) Modified the format of logging with my app. It needs to use this exact format. @cee: {"logsene-app-token": "<your-app-token>", "message": "<your-normal-message>"} The initial "@cee:" is important as it tells Logsene that the format will be JSON. Without that messages won't show up in the dashboard. 2.) Configure the app to log to STDOUT or STDERR. 3.) Create a user provided service for the log drain. Here's what I used. cf cups logsene-logs -l syslog://logsene-receiver-syslog.sematext.com:514 4.) Bind the log drain to my app and deploy the app to CF. I did this with a "services:" entry in the manifest.yml file, but you can also do it with the cf bind-service command. The only drawback is that this only lets me send logging from the app. As you mentioned, there's no way to control the format of the system logging, so messages from the router & dea don't show up in Logsene. Dan - *"Do multiple apps on CF send logs from the same IP address?"* – |
|
|