Re: Log drain for an app


Daniel Mikusa
 

John,

Thanks again! I was able to get this working. Some notes below if anyone
else is trying this out.


I see three questions in your email, which I'll try to address in turn:

- *"Can my application send logs with a unique token?"* – Your
application can add any text it likes to the message, of course. When
they're sent to the syslog drain, the messages will be embedded in a
syslog-formatted line. Looking at Logsene's syslog example
<https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Syslog#Syslog-Example>,
it seems that they expect the syslog message to contain a JSON payload with
the token as a property. If your application produces that JSON, I think it
would be compatible. However, the Loggregator system does not wrap bare
loglines into that format, nor can it be configured to do so without
rewriting code.

This is what worked for me.
1.) Modified the format of logging with my app. It needs to use this exact
format.

@cee: {"logsene-app-token": "<your-app-token>", "message":
"<your-normal-message>"}

The initial "@cee:" is important as it tells Logsene that the format will
be JSON. Without that messages won't show up in the dashboard.

2.) Configure the app to log to STDOUT or STDERR.

3.) Create a user provided service for the log drain. Here's what I used.

cf cups logsene-logs -l syslog://logsene-receiver-syslog.sematext.com:514

4.) Bind the log drain to my app and deploy the app to CF. I did this with
a "services:" entry in the manifest.yml file, but you can also do it with
the cf bind-service command.

The only drawback is that this only lets me send logging from the app. As
you mentioned, there's no way to control the format of the system logging,
so messages from the router & dea don't show up in Logsene.

Dan



- *"Do multiple apps on CF send logs from the same IP address?"* –
Yes. But it's worse than that: not only do multiple app streams come from
the same IP address, but a single application's stream can come from
multiple IP addresses. So this is probably not good from Logsene's point of
view.
- *"Is Loggregator's HTTPS transport compatible with the ElasticSearch
API?"* – No. Loggregator makes a POST request to the HTTPS endpoint by
putting a syslog-formatted line into the body of the request. It does not
have support for building an ElasticSearch-compatible JSON payload around
the message.

It appears to me that the best shot you have of compatibility with Logsene
is having your application build messages in the expected way, with JSON
wrapper (if that's truly needed; my quick read of the syslog example I
linked above was unclear). Keep in mind that Loggregator sends each *line* separately,
so your JSON payload must be a single line to be transmitted correctly.

– John Tuley

On Fri, Jun 12, 2015 at 11:13 AM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

Hello,

I'm looking at sending logs from my app to Logsene [1] and I'm trying to
figure out if this is going to work. From their instructions it seems like
there are two possibilities: syslog & https

I'm not sure syslog will work as Logsene seems to either require a unique
token to be included in the syslog event or to have all syslog traffic from
my app come from one IP. I'm not sure that the first is possible, and the
second won't work as multiple apps on CF could send logs from the same IP
(, please correct me if I'm wrong on either point).

That leaves me with HTTPS. According to their docs, they support the
elasticsearch api [2] through which you can post events to them. It seems
to expect a JSON payload, with a standard format.

I see in the CF docs [3] that we support sending logs via HTTPS but it
doesn't really say how the information is sent via HTTPS. Does anyone know
if this will be compatible? and where I can find more information about
how we send log data via HTTPS?

Thanks

Dan


[1] -
https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Logsene+documentation%3A+Home
[2] -
https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Index+Events+via+Elasticsearch+API
[3] -
http://docs.cloudfoundry.org/devguide/services/log-management.html#config

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.