Re: Doppler/Firehose - Multiline Log Entry

Jim CF Campbell

cf logs
is actually maintained by the CLI team under Dies
<>. You can talk to them.
I'll certainly support you by helping explain the need. I'd think we want a
general solution (token in ENV for instance).

On Tue, Apr 12, 2016 at 11:02 AM, Mike Youngstrom <youngm(a)> wrote:


If I submitted a CLI PR to change the cf logs command to substitute /u2028
with /n could the loggregator team get behind that?


On Tue, Apr 12, 2016 at 10:20 AM, Jim CF Campbell <jcampbell(a)>


When you get a bit more desperate ;-) here is a nozzle plug in
<> for the CLI. It's
attaches to the firehose to display everything, but would be easy to modify
to just look at a single app, and sub out the magic token for newlines.


On Tue, Apr 12, 2016 at 9:56 AM, Mike Youngstrom <youngm(a)>

Hi David,

The problem for me is that I'm searching for a solution that can works
for development (though less of a priority cause you can switch config
between dev and cf) and for viewing logs via "cf logs" in addition to a log
aggregator. I had hoped that /u2028 would work for viewing logs via "cf
logs" but it doesn't in bash. I'd need to write a plugin or something for
cf logs and train all my users to use it. Certainly possible but I'm not
that desperate yet. :)


On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)>

FWIW, the technique is to have your logging solution (eg, logback,
log4j) log a token (eg, \u2028) other than \n to denote line breaks in
your stack traces; and then have your log aggregation software replace that
token with a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else; eg

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)> wrote:

Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution.
Hope you make progress on a metric format available to an app in a
container. I too would like a tracker link to such a feature if there is


On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)>

Hi Jim,

So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.


On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <
jcampbell(a)> wrote:

Hi Mike and Alex,

Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for
Logstash. Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana
will display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

^^^ Seems that passing a string with an actual newline in it is the
only way to make gsub work


to replace the token with a regular newline again so it displays
"properly" in Kibana.



On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)>

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?


On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou | |

Jim Campbell | Product Manager | Cloud Foundry | |

Jim Campbell | Product Manager | Cloud Foundry | |
Jim Campbell | Product Manager | Cloud Foundry | | 303.618.0963

Join { to automatically receive all group messages.