Re: Doppler/Firehose - Multiline Log Entry


David Laing
 

FWIW, the technique is to have your logging solution (eg, logback, log4j)
log a token (eg, \u2028) other than \n to denote line breaks in your stack
traces; and then have your log aggregation software replace that token with
a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else; eg NEWLINE

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com> wrote:

Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution. Hope
you make progress on a metric format available to an app in a container. I
too would like a tracker link to such a feature if there is one.

Thanks,
Mike

On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

Hi Jim,

So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.

Mike

On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

Hi Mike and Alex,

Two things - for Java, we are working toward defining an enhanced metric
format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for Logstash.
Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana will
display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

"]
^^^ Seems that passing a string with an actual newline in it is the only
way to make gsub work

}

to replace the token with a regular newline again so it displays
"properly" in Kibana.

[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>

[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>


On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?

Mike

On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How correctly
deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.