Re: Log drain for an app


John Tuley <jtuley@...>
 

Dan,

I see three questions in your email, which I'll try to address in turn:

- *"Can my application send logs with a unique token?"* – Your
application can add any text it likes to the message, of course. When
they're sent to the syslog drain, the messages will be embedded in a
syslog-formatted line. Looking at Logsene's syslog example
<https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Syslog#Syslog-Example>,
it seems that they expect the syslog message to contain a JSON payload with
the token as a property. If your application produces that JSON, I think it
would be compatible. However, the Loggregator system does not wrap bare
loglines into that format, nor can it be configured to do so without
rewriting code.
- *"Do multiple apps on CF send logs from the same IP address?"* – Yes.
But it's worse than that: not only do multiple app streams come from the
same IP address, but a single application's stream can come from multiple
IP addresses. So this is probably not good from Logsene's point of view.
- *"Is Loggregator's HTTPS transport compatible with the ElasticSearch
API?"* – No. Loggregator makes a POST request to the HTTPS endpoint by
putting a syslog-formatted line into the body of the request. It does not
have support for building an ElasticSearch-compatible JSON payload around
the message.

It appears to me that the best shot you have of compatibility with Logsene
is having your application build messages in the expected way, with JSON
wrapper (if that's truly needed; my quick read of the syslog example I
linked above was unclear). Keep in mind that Loggregator sends each
*line* separately,
so your JSON payload must be a single line to be transmitted correctly.

– John Tuley

On Fri, Jun 12, 2015 at 11:13 AM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

Hello,

I'm looking at sending logs from my app to Logsene [1] and I'm trying to
figure out if this is going to work. From their instructions it seems like
there are two possibilities: syslog & https

I'm not sure syslog will work as Logsene seems to either require a unique
token to be included in the syslog event or to have all syslog traffic from
my app come from one IP. I'm not sure that the first is possible, and the
second won't work as multiple apps on CF could send logs from the same IP
(, please correct me if I'm wrong on either point).

That leaves me with HTTPS. According to their docs, they support the
elasticsearch api [2] through which you can post events to them. It seems
to expect a JSON payload, with a standard format.

I see in the CF docs [3] that we support sending logs via HTTPS but it
doesn't really say how the information is sent via HTTPS. Does anyone know
if this will be compatible? and where I can find more information about
how we send log data via HTTPS?

Thanks

Dan


[1] -
https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Logsene+documentation%3A+Home
[2] -
https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Index+Events+via+Elasticsearch+API
[3] -
http://docs.cloudfoundry.org/devguide/services/log-management.html#config

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.