Re: Doppler/Firehose - Multiline Log Entry

Mike Youngstrom <youngm@...>

Hi David,

The problem for me is that I'm searching for a solution that can works for
development (though less of a priority cause you can switch config between
dev and cf) and for viewing logs via "cf logs" in addition to a log
aggregator. I had hoped that /u2028 would work for viewing logs via "cf
logs" but it doesn't in bash. I'd need to write a plugin or something for
cf logs and train all my users to use it. Certainly possible but I'm not
that desperate yet. :)


On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)> wrote:

FWIW, the technique is to have your logging solution (eg, logback, log4j)
log a token (eg, \u2028) other than \n to denote line breaks in
your stack traces; and then have your log aggregation software replace that
token with a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else; eg NEWLINE

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)> wrote:

Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution.
Hope you make progress on a metric format available to an app in a
container. I too would like a tracker link to such a feature if there is


On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)>

Hi Jim,

So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.


On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <jcampbell(a)>

Hi Mike and Alex,

Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for Logstash.
Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana will
display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

^^^ Seems that passing a string with an actual newline in it is the
only way to make gsub work


to replace the token with a regular newline again so it displays
"properly" in Kibana.



On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)>

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?


On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou | |

Jim Campbell | Product Manager | Cloud Foundry | |

Join { to automatically receive all group messages.