Date   

Re: Doppler/Firehose - Multiline Log Entry

Jim CF Campbell
 

That strategy is going to be hard to sell. Diego's Executor takes the log
lines out of Garden and drops them into dropsonde messages. I doubt they'll
think it's a good idea to implement substitution in that processing. You
can certainly ask Eric - he's very aware of the underlying problem.

After that point, the Loggregator system does it's best to touch messages
as little as possible, and to improve performance and reliability, we have
thinking about the future that will lower the amount of touching ever
further. The next place that log message processing can be done is either
in a nozzle, or the injester of a log aggregator.

I'd vote for those downstream places - a single configuration and
algorithm instead of distributed across runner VMs.

On Tue, Apr 12, 2016 at 2:15 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

I was thinking whoever demarcates and submits the original event to
loggregator. dea_logging_agent and the equivalent in Deigo. Doing it at
that point could provide a bit more flexibility. I know this isn't
necessarily the loggregator team's code but I think loggregator team buy
off would be important for those projects to accept such a PR.

Unless you can think of a better place to make that transformation within
the loggregator processing chain?

Mike

On Tue, Apr 12, 2016 at 2:02 PM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

what exactly do you mean by "event creation time"?

On Tue, Apr 12, 2016 at 1:57 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Before I submit the CLI issue let me ask one more question.

Would it be better to replace the newline token with /n at event
creation time instead of asking the cli, splunk, anyone listening on the
firehose, etc. to do so?

The obvious downside is this would probably need to be a global
configuration. However, I know my organization wouldn't have a problem
swapping /u2028 with /n for a deployment. The feature would obviously be
off by default.

Thoughs?

Mike

On Tue, Apr 12, 2016 at 11:24 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Sounds good. I'll submit an issue to start the discussion. I imagine
the first question Dies will ask though is if you would support something
like that. :)

Mike

On Tue, Apr 12, 2016 at 11:12 AM, Jim CF Campbell <jcampbell(a)pivotal.io
wrote:
cf logs
<https://github.com/cloudfoundry/cli/blob/40eb5be48eaac697c3700d5f1e6f654bae471cec/cf/commands/application/logs.go>
is actually maintained by the CLI team under Dies
<https://www.pivotaltracker.com/n/projects/892938>. You can talk to
them. I'll certainly support you by helping explain the need. I'd think we
want a general solution (token in ENV for instance).



On Tue, Apr 12, 2016 at 11:02 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Jim,

If I submitted a CLI PR to change the cf logs command to substitute
/u2028 with /n could the loggregator team get behind that?

Mike

On Tue, Apr 12, 2016 at 10:20 AM, Jim CF Campbell <
jcampbell(a)pivotal.io> wrote:

Mike,

When you get a bit more desperate ;-) here is a nozzle plug in
<https://github.com/jtuchscherer/nozzle-plugin> for the CLI. It's
attaches to the firehose to display everything, but would be easy to modify
to just look at a single app, and sub out the magic token for newlines.

Jim

On Tue, Apr 12, 2016 at 9:56 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi David,

The problem for me is that I'm searching for a solution that can
works for development (though less of a priority cause you can switch
config between dev and cf) and for viewing logs via "cf logs" in addition
to a log aggregator. I had hoped that /u2028 would work for viewing logs
via "cf logs" but it doesn't in bash. I'd need to write a plugin or
something for cf logs and train all my users to use it. Certainly possible
but I'm not that desperate yet. :)

Mike

On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com>
wrote:

FWIW, the technique is to have your logging solution (eg, logback,
log4j) log a token (eg, \u2028) other than \n to denote line
breaks in your stack traces; and then have your log aggregation software
replace that token with a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else;
eg NEWLINE

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com>
wrote:

Finally got around to testing this. Preliminary testing show
that "\u2028" doesn't function as a new line character in bash
and causes eclipse console to wig out. I don't think "\u2028"
is a viable long term solution. Hope you make progress on a metric format
available to an app in a container. I too would like a tracker link to
such a feature if there is one.

Thanks,
Mike

On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <
youngm(a)gmail.com> wrote:

Hi Jim,

So, to be clear what we're basically doing is using unicode
newline character to fool loggregator (which is looking for \n) into
thinking that it isn't a new log event right? Does \u2028 work as a new
line character when tailing logs in the CLI? Anyone tried this unicode new
line character in various consoles? IDE, xterm, etc? I'm wondering if
developers will need to have different config for development.

Mike

On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <
jcampbell(a)pivotal.io> wrote:

Hi Mike and Alex,

Two things - for Java, we are working toward defining an
enhanced metric format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for
Logstash. Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which
Kibana will display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

"]
^^^ Seems that passing a string with an actual newline in it is
the only way to make gsub work

}

to replace the token with a regular newline again so it
displays "properly" in Kibana.

[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>

[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>


On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <
youngm(a)gmail.com> wrote:

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?

Mike

On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Re: Doppler/Firehose - Multiline Log Entry

Mike Youngstrom
 

I was thinking whoever demarcates and submits the original event to
loggregator. dea_logging_agent and the equivalent in Deigo. Doing it at
that point could provide a bit more flexibility. I know this isn't
necessarily the loggregator team's code but I think loggregator team buy
off would be important for those projects to accept such a PR.

Unless you can think of a better place to make that transformation within
the loggregator processing chain?

Mike

On Tue, Apr 12, 2016 at 2:02 PM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

what exactly do you mean by "event creation time"?

On Tue, Apr 12, 2016 at 1:57 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

Before I submit the CLI issue let me ask one more question.

Would it be better to replace the newline token with /n at event creation
time instead of asking the cli, splunk, anyone listening on the firehose,
etc. to do so?

The obvious downside is this would probably need to be a global
configuration. However, I know my organization wouldn't have a problem
swapping /u2028 with /n for a deployment. The feature would obviously be
off by default.

Thoughs?

Mike

On Tue, Apr 12, 2016 at 11:24 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Sounds good. I'll submit an issue to start the discussion. I imagine
the first question Dies will ask though is if you would support something
like that. :)

Mike

On Tue, Apr 12, 2016 at 11:12 AM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

cf logs
<https://github.com/cloudfoundry/cli/blob/40eb5be48eaac697c3700d5f1e6f654bae471cec/cf/commands/application/logs.go>
is actually maintained by the CLI team under Dies
<https://www.pivotaltracker.com/n/projects/892938>. You can talk to
them. I'll certainly support you by helping explain the need. I'd think we
want a general solution (token in ENV for instance).



On Tue, Apr 12, 2016 at 11:02 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Jim,

If I submitted a CLI PR to change the cf logs command to substitute
/u2028 with /n could the loggregator team get behind that?

Mike

On Tue, Apr 12, 2016 at 10:20 AM, Jim CF Campbell <
jcampbell(a)pivotal.io> wrote:

Mike,

When you get a bit more desperate ;-) here is a nozzle plug in
<https://github.com/jtuchscherer/nozzle-plugin> for the CLI. It's
attaches to the firehose to display everything, but would be easy to modify
to just look at a single app, and sub out the magic token for newlines.

Jim

On Tue, Apr 12, 2016 at 9:56 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi David,

The problem for me is that I'm searching for a solution that can
works for development (though less of a priority cause you can switch
config between dev and cf) and for viewing logs via "cf logs" in addition
to a log aggregator. I had hoped that /u2028 would work for viewing logs
via "cf logs" but it doesn't in bash. I'd need to write a plugin or
something for cf logs and train all my users to use it. Certainly possible
but I'm not that desperate yet. :)

Mike

On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com>
wrote:

FWIW, the technique is to have your logging solution (eg, logback,
log4j) log a token (eg, \u2028) other than \n to denote line
breaks in your stack traces; and then have your log aggregation software
replace that token with a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else; eg
NEWLINE

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com>
wrote:

Finally got around to testing this. Preliminary testing show that
"\u2028" doesn't function as a new line character in bash and
causes eclipse console to wig out. I don't think "\u2028" is a
viable long term solution. Hope you make progress on a metric format
available to an app in a container. I too would like a tracker link to
such a feature if there is one.

Thanks,
Mike

On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com
wrote:
Hi Jim,

So, to be clear what we're basically doing is using unicode
newline character to fool loggregator (which is looking for \n) into
thinking that it isn't a new log event right? Does \u2028 work as a new
line character when tailing logs in the CLI? Anyone tried this unicode new
line character in various consoles? IDE, xterm, etc? I'm wondering if
developers will need to have different config for development.

Mike

On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <
jcampbell(a)pivotal.io> wrote:

Hi Mike and Alex,

Two things - for Java, we are working toward defining an
enhanced metric format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for
Logstash. Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which
Kibana will display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

"]
^^^ Seems that passing a string with an actual newline in it is
the only way to make gsub work

}

to replace the token with a regular newline again so it displays
"properly" in Kibana.

[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>

[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>


On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <
youngm(a)gmail.com> wrote:

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?

Mike

On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Re: Doppler/Firehose - Multiline Log Entry

Jim CF Campbell
 

what exactly do you mean by "event creation time"?

On Tue, Apr 12, 2016 at 1:57 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

Before I submit the CLI issue let me ask one more question.

Would it be better to replace the newline token with /n at event creation
time instead of asking the cli, splunk, anyone listening on the firehose,
etc. to do so?

The obvious downside is this would probably need to be a global
configuration. However, I know my organization wouldn't have a problem
swapping /u2028 with /n for a deployment. The feature would obviously be
off by default.

Thoughs?

Mike

On Tue, Apr 12, 2016 at 11:24 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Sounds good. I'll submit an issue to start the discussion. I imagine
the first question Dies will ask though is if you would support something
like that. :)

Mike

On Tue, Apr 12, 2016 at 11:12 AM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

cf logs
<https://github.com/cloudfoundry/cli/blob/40eb5be48eaac697c3700d5f1e6f654bae471cec/cf/commands/application/logs.go>
is actually maintained by the CLI team under Dies
<https://www.pivotaltracker.com/n/projects/892938>. You can talk to
them. I'll certainly support you by helping explain the need. I'd think we
want a general solution (token in ENV for instance).



On Tue, Apr 12, 2016 at 11:02 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Jim,

If I submitted a CLI PR to change the cf logs command to substitute
/u2028 with /n could the loggregator team get behind that?

Mike

On Tue, Apr 12, 2016 at 10:20 AM, Jim CF Campbell <jcampbell(a)pivotal.io
wrote:
Mike,

When you get a bit more desperate ;-) here is a nozzle plug in
<https://github.com/jtuchscherer/nozzle-plugin> for the CLI. It's
attaches to the firehose to display everything, but would be easy to modify
to just look at a single app, and sub out the magic token for newlines.

Jim

On Tue, Apr 12, 2016 at 9:56 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi David,

The problem for me is that I'm searching for a solution that can
works for development (though less of a priority cause you can switch
config between dev and cf) and for viewing logs via "cf logs" in addition
to a log aggregator. I had hoped that /u2028 would work for viewing logs
via "cf logs" but it doesn't in bash. I'd need to write a plugin or
something for cf logs and train all my users to use it. Certainly possible
but I'm not that desperate yet. :)

Mike

On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com>
wrote:

FWIW, the technique is to have your logging solution (eg, logback,
log4j) log a token (eg, \u2028) other than \n to denote line breaks
in your stack traces; and then have your log aggregation software replace
that token with a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else; eg
NEWLINE

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com>
wrote:

Finally got around to testing this. Preliminary testing show that "
\u2028" doesn't function as a new line character in bash and
causes eclipse console to wig out. I don't think "\u2028" is a
viable long term solution. Hope you make progress on a metric format
available to an app in a container. I too would like a tracker link to
such a feature if there is one.

Thanks,
Mike

On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi Jim,

So, to be clear what we're basically doing is using unicode
newline character to fool loggregator (which is looking for \n) into
thinking that it isn't a new log event right? Does \u2028 work as a new
line character when tailing logs in the CLI? Anyone tried this unicode new
line character in various consoles? IDE, xterm, etc? I'm wondering if
developers will need to have different config for development.

Mike

On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <
jcampbell(a)pivotal.io> wrote:

Hi Mike and Alex,

Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for
Logstash. Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which
Kibana will display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

"]
^^^ Seems that passing a string with an actual newline in it is
the only way to make gsub work

}

to replace the token with a regular newline again so it displays
"properly" in Kibana.

[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>

[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>


On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <
youngm(a)gmail.com> wrote:

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?

Mike

On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Re: Doppler/Firehose - Multiline Log Entry

Mike Youngstrom
 

Before I submit the CLI issue let me ask one more question.

Would it be better to replace the newline token with /n at event creation
time instead of asking the cli, splunk, anyone listening on the firehose,
etc. to do so?

The obvious downside is this would probably need to be a global
configuration. However, I know my organization wouldn't have a problem
swapping /u2028 with /n for a deployment. The feature would obviously be
off by default.

Thoughs?

Mike

On Tue, Apr 12, 2016 at 11:24 AM, Mike Youngstrom <youngm(a)gmail.com> wrote:

Sounds good. I'll submit an issue to start the discussion. I imagine the
first question Dies will ask though is if you would support something like
that. :)

Mike

On Tue, Apr 12, 2016 at 11:12 AM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

cf logs
<https://github.com/cloudfoundry/cli/blob/40eb5be48eaac697c3700d5f1e6f654bae471cec/cf/commands/application/logs.go>
is actually maintained by the CLI team under Dies
<https://www.pivotaltracker.com/n/projects/892938>. You can talk to
them. I'll certainly support you by helping explain the need. I'd think we
want a general solution (token in ENV for instance).



On Tue, Apr 12, 2016 at 11:02 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Jim,

If I submitted a CLI PR to change the cf logs command to substitute
/u2028 with /n could the loggregator team get behind that?

Mike

On Tue, Apr 12, 2016 at 10:20 AM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

Mike,

When you get a bit more desperate ;-) here is a nozzle plug in
<https://github.com/jtuchscherer/nozzle-plugin> for the CLI. It's
attaches to the firehose to display everything, but would be easy to modify
to just look at a single app, and sub out the magic token for newlines.

Jim

On Tue, Apr 12, 2016 at 9:56 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi David,

The problem for me is that I'm searching for a solution that can works
for development (though less of a priority cause you can switch config
between dev and cf) and for viewing logs via "cf logs" in addition to a log
aggregator. I had hoped that /u2028 would work for viewing logs via "cf
logs" but it doesn't in bash. I'd need to write a plugin or something for
cf logs and train all my users to use it. Certainly possible but I'm not
that desperate yet. :)

Mike

On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com>
wrote:

FWIW, the technique is to have your logging solution (eg, logback,
log4j) log a token (eg, \u2028) other than \n to denote line breaks
in your stack traces; and then have your log aggregation software replace
that token with a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else; eg
NEWLINE

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com>
wrote:

Finally got around to testing this. Preliminary testing show that "
\u2028" doesn't function as a new line character in bash and causes
eclipse console to wig out. I don't think "\u2028" is a viable
long term solution. Hope you make progress on a metric format available to
an app in a container. I too would like a tracker link to such a feature
if there is one.

Thanks,
Mike

On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi Jim,

So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.

Mike

On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <
jcampbell(a)pivotal.io> wrote:

Hi Mike and Alex,

Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for
Logstash. Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana
will display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

"]
^^^ Seems that passing a string with an actual newline in it is
the only way to make gsub work

}

to replace the token with a regular newline again so it displays
"properly" in Kibana.

[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>

[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>


On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <
youngm(a)gmail.com> wrote:

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?

Mike

On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963


vcap.component.discover in 233

Matt Cholick
 

We recently upgrade from 228 to 233. Since then, one thing I'm seeing is
that neither DEA nor HM9000 is responding to a vcap.component.discover on
nats. Is this expected? I've dug a bit and haven't found a commit yet.
Wondering if the bug is in our environment/config or our expectations.

CloudController and etc are examples of some things that still respond to
this request.

-Matt Cholick


Re: Doppler/Firehose - Multiline Log Entry

Mike Youngstrom
 

Sounds good. I'll submit an issue to start the discussion. I imagine the
first question Dies will ask though is if you would support something like
that. :)

Mike

On Tue, Apr 12, 2016 at 11:12 AM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

cf logs
<https://github.com/cloudfoundry/cli/blob/40eb5be48eaac697c3700d5f1e6f654bae471cec/cf/commands/application/logs.go>
is actually maintained by the CLI team under Dies
<https://www.pivotaltracker.com/n/projects/892938>. You can talk to them.
I'll certainly support you by helping explain the need. I'd think we want a
general solution (token in ENV for instance).



On Tue, Apr 12, 2016 at 11:02 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Jim,

If I submitted a CLI PR to change the cf logs command to substitute
/u2028 with /n could the loggregator team get behind that?

Mike

On Tue, Apr 12, 2016 at 10:20 AM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

Mike,

When you get a bit more desperate ;-) here is a nozzle plug in
<https://github.com/jtuchscherer/nozzle-plugin> for the CLI. It's
attaches to the firehose to display everything, but would be easy to modify
to just look at a single app, and sub out the magic token for newlines.

Jim

On Tue, Apr 12, 2016 at 9:56 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi David,

The problem for me is that I'm searching for a solution that can works
for development (though less of a priority cause you can switch config
between dev and cf) and for viewing logs via "cf logs" in addition to a log
aggregator. I had hoped that /u2028 would work for viewing logs via "cf
logs" but it doesn't in bash. I'd need to write a plugin or something for
cf logs and train all my users to use it. Certainly possible but I'm not
that desperate yet. :)

Mike

On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com>
wrote:

FWIW, the technique is to have your logging solution (eg, logback,
log4j) log a token (eg, \u2028) other than \n to denote line breaks
in your stack traces; and then have your log aggregation software replace
that token with a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else; eg
NEWLINE

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com> wrote:

Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution.
Hope you make progress on a metric format available to an app in a
container. I too would like a tracker link to such a feature if there is
one.

Thanks,
Mike

On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi Jim,

So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.

Mike

On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <
jcampbell(a)pivotal.io> wrote:

Hi Mike and Alex,

Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for
Logstash. Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana
will display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

"]
^^^ Seems that passing a string with an actual newline in it is the
only way to make gsub work

}

to replace the token with a regular newline again so it displays
"properly" in Kibana.

[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>

[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>


On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)gmail.com
wrote:
I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?

Mike

On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Re: Doppler/Firehose - Multiline Log Entry

Jim CF Campbell
 

cf logs
<https://github.com/cloudfoundry/cli/blob/40eb5be48eaac697c3700d5f1e6f654bae471cec/cf/commands/application/logs.go>
is actually maintained by the CLI team under Dies
<https://www.pivotaltracker.com/n/projects/892938>. You can talk to them.
I'll certainly support you by helping explain the need. I'd think we want a
general solution (token in ENV for instance).

On Tue, Apr 12, 2016 at 11:02 AM, Mike Youngstrom <youngm(a)gmail.com> wrote:

Jim,

If I submitted a CLI PR to change the cf logs command to substitute /u2028
with /n could the loggregator team get behind that?

Mike

On Tue, Apr 12, 2016 at 10:20 AM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

Mike,

When you get a bit more desperate ;-) here is a nozzle plug in
<https://github.com/jtuchscherer/nozzle-plugin> for the CLI. It's
attaches to the firehose to display everything, but would be easy to modify
to just look at a single app, and sub out the magic token for newlines.

Jim

On Tue, Apr 12, 2016 at 9:56 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi David,

The problem for me is that I'm searching for a solution that can works
for development (though less of a priority cause you can switch config
between dev and cf) and for viewing logs via "cf logs" in addition to a log
aggregator. I had hoped that /u2028 would work for viewing logs via "cf
logs" but it doesn't in bash. I'd need to write a plugin or something for
cf logs and train all my users to use it. Certainly possible but I'm not
that desperate yet. :)

Mike

On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com>
wrote:

FWIW, the technique is to have your logging solution (eg, logback,
log4j) log a token (eg, \u2028) other than \n to denote line breaks in
your stack traces; and then have your log aggregation software replace that
token with a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else; eg
NEWLINE

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com> wrote:

Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution.
Hope you make progress on a metric format available to an app in a
container. I too would like a tracker link to such a feature if there is
one.

Thanks,
Mike

On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi Jim,

So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.

Mike

On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <
jcampbell(a)pivotal.io> wrote:

Hi Mike and Alex,

Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for
Logstash. Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana
will display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

"]
^^^ Seems that passing a string with an actual newline in it is the
only way to make gsub work

}

to replace the token with a regular newline again so it displays
"properly" in Kibana.

[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>

[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>


On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?

Mike

On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Re: Request for Multibuildpack Use Cases

Mike Youngstrom
 

John,

It sounds like the buildpack team is thinking the multi buildpack feature
would only work for buildpacks they provide not a custom
"dependency-resolution" buildpack. Or at least that is how I understood
the message from Danny Rosen earlier in the thread.

Mike

On Tue, Apr 12, 2016 at 10:45 AM, John Feminella <jxf(a)pivotal.io> wrote:

Multibuildpack is absolutely useful and I'm excited for this proposal.

I encounter a lot of use cases for this. The most common is that an
application wants to pull in private dependencies during a future
dependency-resolution step of a later buildpack, but the dependency
resolver needs to be primed in some specific way. If you wait until
buildpack time it's too late.

On Heroku, for example, this is accomplished by having something like the
netrc buildpack (
https://github.com/timshadel/heroku-buildpack-github-netrc), adding a
GITHUB_TOKEN environment variable, and then running your "real" buildpack.
The netrc BP runs first, allowing Bundler to see the private dependencies.

best,
~ jf

On Tue, Apr 12, 2016 at 12:36 PM Jack Cai <greensight(a)gmail.com> wrote:

It would be more useful if the multi-buildpack can reference an admin
buildpack in addition to a remote git-hosted buildpack. :-)

Jack


On Tue, Apr 12, 2016 at 6:38 AM, David Illsley <davidillsley(a)gmail.com>
wrote:

In the past we've used the multi-buildpack to be able to use ruby sass
to compile SCSS for non-ruby projects (node and Java). In that case we used
the multi-buildpack and a .buildpacks file which worked reasonably well
(and was very clear).

On Mon, Apr 11, 2016 at 1:15 AM, Danny Rosen <drosen(a)pivotal.io> wrote:

Hi there,

The CF Buildpacks team is considering taking on a line of work to
provide more formal support for multibuildpacks. Before we start, we would
be interested in learning if any community users have compelling use cases
they could share with us.

For more information on multibuildpacks, see Heroku's documentation [1]

[1] -
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app


Re: [cf-bosh] Re: Re: [Metron Agent] failed to generate job templates with metron agent on top of OpenStack Dynamic network

Jim CF Campbell
 

We did a spike <https://www.pivotaltracker.com/story/show/116805009> to
figure out the Metron behavior. We have a story
<https://www.pivotaltracker.com/story/show/117257489> to fix in our icebox,
but not yet sure how that will work.

On Mon, Apr 11, 2016 at 11:24 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

This will not work with dynamic networks. Many jobs in cf-release rely on
data from BOSH to determine their IP so that configuration files can be
rendered up-front by the director rather than at runtime, requiring system
calls to determine IP. metron_agent is one such job, and it tends to be
colocated with each other job (it is what allows all system component logs
to be aggregated through the loggregator system), so this would require all
Cloud Foundry VMs to be on a manual network. You don't need to manually
pick the IPs, you just need to tell BOSH which IPs in the network not to
use and specify these in the "reserved" range.

Since so many different components depend on being able to determine their
IP via BOSH data, there's no quick workaround if you want to stick to using
dynamic networks, but we're aware of this current limitation.

Best,
Amit

On Mon, Apr 11, 2016 at 7:23 PM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote:

Is it a bug of CF or Bosh ?

On Fri, Apr 8, 2016 at 12:08 PM, Ben R <vagcom.ben(a)gmail.com> wrote:

I have the same issue. It has to do with every release since bosh 248.
However, dynamic networks with older bosh releases + cf-231/cf-231 work.

This must be a bug.

Ben R


On Thu, Apr 7, 2016 at 8:55 PM, Yitao Jiang <jiangyt.cn(a)gmail.com>
wrote:

Hi,guys

When deploy CF on top of OpenStack with dynamic network, the jobs
failed with metron-agent
Error filling in template 'syslog_forwarder.conf.erb' (line 44:
undefined method `strip' for nil:NilClass)

here's related logs

​D​
etecting deployment changes
----------------------------
Releases
cf
version type changed: String -> Fixnum
- 233
+ 233

Compilation
No changes

Update
± canaries:
- 1
+ 0

Resource pools
No changes

Disk pools
No changes

Networks
dynamic-net
+ name: dynamic-net
subnets
10.0.0.0/24
cloud_properties
+ net_id: 0700ae03-4b38-464e-b40d-0a9c8dd18ff0
+ security_groups: ["Test OS SG_20160128T070152Z"]
+ dns: ["114.114.114.114", "8.8.8.8"]

+ range: 10.0.0.0/24
+ name: Test OS Sub Internal Network_20160128T070152Z

+ type: dynamic


Jobs
stats_z1

± networks:

- {"name"=>"cf1"}

+ {"name"=>"dynamic-net"}


Properties
No changes


Meta
No changes


Please review all changes carefully

Deploying
---------

Are you sure you want to deploy? (type 'yes' to continue): yes


Director task 57
Started preparing deployment > Preparing deployment. Done (00:00:03)


Error 100: Unable to render instance groups for deployment. Errors
are:
- Unable to render jobs for instance group 'stats_z1'. Errors are:

- Unable to render templates for job 'metron_agent'. Errors are:

- Error filling in template 'syslog_forwarder.conf.erb' (line
44: undefined method `strip' for nil:NilClass)

Task 57 error

For a more detailed error report, run: bosh task 57 --debug
as the ip manged by OpenStack, bosh cannot get the actual ip address of
each vm until vm alive, this lead to the generated job spec doesn't contain
ip address infos
so, must i have to configure network type to manual?​

snippets of deployment yml

1001 - name: dynamic-net
1002 subnets:
1003 - cloud_properties:
1004 net_id: 0700ae03-4b38-464e-b40d-0a9c8dd18ff0
1005 security_groups:
1006 - Test OS SG_20160128T070152Z
1007 dns:
1010 - 114.114.114.114
1011 - 8.8.8.8
1012 range: 10.0.0.0/24
1013 name: Test OS Sub Internal Network_20160128T070152Z
1014 type: dynamic

​Rendered job spec

{"deployment"=>"staging-01", "job"=

{"name"=>"stats_z1", "templates"=>[{"name"=>"collector",
"version"=>"6c210292f18d129e9a037fe7053836db2d494344",
"sha1"=>"38927f47b15c2daf6c8a2e7c760e73e5ff90
dfd4", "blobstore_id"=>"23531029-0ee1-4267-8863-b5f931afaecb"},
{"name"=>"metron_agent",
"version"=>"2b80a211127fc642fc8bb0d14d7eb30c37730db3", "sha1"=>"150f2
7445c2ef960951c1f26606525d41ec629b2",
"blobstore_id"=>"e87174dc-f3f7-4768-94cd-74f299813528"}],
"template"=>"collector", "version"=>"6c210292f18d129e9a037fe70
53836db2d494344", "sha1"=>"38927f47b15c2daf6c8a2e7c760e73e5ff90dfd4",
"blobstore_id"=>"23531029-0ee1-4267-8863-b5f931afaecb"}, "index"=>0,
"bootstrap"=>true,
"name"=>"stats_z1", "id"=>"99f349d0-fb5d-4de7-9912-3de5559d2f19",
"az"=>nil,

*"networks"=>{"dynamic-net"=>{"type"=>"dynamic",
"cloud_properties"=>{"net_id"=>"0700ae03-4b38-464e-b40d-0a9c8dd18ff0",
"security_groups"=>["Test OS SG_20160128T070152Z"]},
"dns"=>["114.114.114.114", "8.8.8.8", "10.0.0.13"], "default"=>["dns",
"gateway"],
"dns_record_name"=>"0.stats-z1.dynamic-net.staging-01.microbosh"}}*,
"properties"=>{"collector"=>{"aws"=>{
"access_key_id"=>nil, "secret_access_key"=>nil},
"datadog"=>{"api_key"=>nil, "application_key"=>nil},
"deployment_name"=>nil, "logging_level"=>"info", "interv
als"=>{"discover"=>60, "healthz"=>30, "local_metrics"=>30,
"nats_ping"=>30, "prune"=>300, "varz"=>30}, "use_aws_cloudwatch"=>false,
"use_datadog"=>false, "use
_tsdb"=>false, "opentsdb"=>{"address"=>nil, "port"=>nil},
"use_graphite"=>false, "graphite"=>{"address"=>nil, "port"=>nil},
"memory_threshold"=>800}, "nats"=>
{"machines"=>["10.0.0.127"], "password"=>"NATS_PASSWORD",
"port"=>4222, "user"=>"NATS_USER"},
"syslog_daemon_config"=>{"address"=>nil, "port"=>nil, "transport
"=>"tcp", "fallback_addresses"=>[], "custom_rule"=>"",
"max_message_size"=>"4k"},
"metron_agent"=>{"dropsonde_incoming_port"=>3457, "preferred_protocol"=>"udp
", "tls"=>{"client_cert"=>"", "client_key"=>""}, "debug"=>false,
"zone"=>"z1", "deployment"=>"ya-staging-01",
"tcp"=>{"batching_buffer_bytes"=>10240, "batchin
g_buffer_flush_interval_milliseconds"=>100},
"logrotate"=>{"freq_min"=>5, "rotate"=>7, "size"=>"50M"},
"buffer_size"=>10000, "enable_buffer"=>false}, "metron_
endpoint"=>{"shared_secret"=>"LOGGREGATOR_ENDPOINT_SHARED_SECRET"},
"loggregator"=>{"tls"=>{"ca_cert"=>""}, "dropsonde_incoming_port"=>3457,
"etcd"=>{"machine
s"=>["10.0.0.133"], "maxconcurrentrequests"=>10}}},
"dns_domain_name"=>"microbosh", "links"=>{},
"address"=>"99f349d0-fb5d-4de7-9912-3de5559d2f19.stats-z1.dyn
amic-net.ya-staging-01.microbosh", "persistent_disk"=>0,
"resource_pool"=>"small_z1"}​

--

Regards,

Yitao

--

Regards,

Yitao
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Re: Request for Multibuildpack Use Cases

John Feminella <jxf@...>
 

Another common use case for me is that an application is polyglot in some
way. For example, you have a blog you'd like to deploy that uses a static
site generator written in (say) Go. You would have the Go buildpack run to
bootstrap and generate the resulting files in some output directory, and
then the staticfile buildpack runs and serves the files from the output
directory.

best,
~ jf

On Tue, Apr 12, 2016 at 12:45 PM John Feminella <jxf(a)pivotal.io> wrote:

Multibuildpack is absolutely useful and I'm excited for this proposal.

I encounter a lot of use cases for this. The most common is that an
application wants to pull in private dependencies during a future
dependency-resolution step of a later buildpack, but the dependency
resolver needs to be primed in some specific way. If you wait until
buildpack time it's too late.

On Heroku, for example, this is accomplished by having something like the
netrc buildpack (
https://github.com/timshadel/heroku-buildpack-github-netrc), adding a
GITHUB_TOKEN environment variable, and then running your "real" buildpack.
The netrc BP runs first, allowing Bundler to see the private dependencies.

best,
~ jf

On Tue, Apr 12, 2016 at 12:36 PM Jack Cai <greensight(a)gmail.com> wrote:

It would be more useful if the multi-buildpack can reference an admin
buildpack in addition to a remote git-hosted buildpack. :-)

Jack


On Tue, Apr 12, 2016 at 6:38 AM, David Illsley <davidillsley(a)gmail.com>
wrote:

In the past we've used the multi-buildpack to be able to use ruby sass
to compile SCSS for non-ruby projects (node and Java). In that case we used
the multi-buildpack and a .buildpacks file which worked reasonably well
(and was very clear).

On Mon, Apr 11, 2016 at 1:15 AM, Danny Rosen <drosen(a)pivotal.io> wrote:

Hi there,

The CF Buildpacks team is considering taking on a line of work to
provide more formal support for multibuildpacks. Before we start, we would
be interested in learning if any community users have compelling use cases
they could share with us.

For more information on multibuildpacks, see Heroku's documentation [1]

[1] -
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app


Re: Doppler/Firehose - Multiline Log Entry

Mike Youngstrom
 

Jim,

If I submitted a CLI PR to change the cf logs command to substitute /u2028
with /n could the loggregator team get behind that?

Mike

On Tue, Apr 12, 2016 at 10:20 AM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

Mike,

When you get a bit more desperate ;-) here is a nozzle plug in
<https://github.com/jtuchscherer/nozzle-plugin> for the CLI. It's
attaches to the firehose to display everything, but would be easy to modify
to just look at a single app, and sub out the magic token for newlines.

Jim

On Tue, Apr 12, 2016 at 9:56 AM, Mike Youngstrom <youngm(a)gmail.com> wrote:

Hi David,

The problem for me is that I'm searching for a solution that can works
for development (though less of a priority cause you can switch config
between dev and cf) and for viewing logs via "cf logs" in addition to a log
aggregator. I had hoped that /u2028 would work for viewing logs via "cf
logs" but it doesn't in bash. I'd need to write a plugin or something for
cf logs and train all my users to use it. Certainly possible but I'm not
that desperate yet. :)

Mike

On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com>
wrote:

FWIW, the technique is to have your logging solution (eg, logback,
log4j) log a token (eg, \u2028) other than \n to denote line breaks in
your stack traces; and then have your log aggregation software replace that
token with a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else; eg
NEWLINE

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com> wrote:

Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution.
Hope you make progress on a metric format available to an app in a
container. I too would like a tracker link to such a feature if there is
one.

Thanks,
Mike

On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi Jim,

So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.

Mike

On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <
jcampbell(a)pivotal.io> wrote:

Hi Mike and Alex,

Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for
Logstash. Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana
will display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

"]
^^^ Seems that passing a string with an actual newline in it is the
only way to make gsub work

}

to replace the token with a regular newline again so it displays
"properly" in Kibana.

[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>

[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>


On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?

Mike

On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Re: dynamic networks

Amit Kumar Gupta
 

That other thread Rob and Alvaro just mentioned currently leaves open the
question of why metron_agent worked when deployed from Micro BOSH on
stemcell 3160 but started giving this error message on 3163. For sanity, I
wlll continue the discussion on that other thread.

Best,
Amit

On Tue, Apr 12, 2016 at 9:40 AM, Release Integration <cfruntime(a)gmail.com>
wrote:

As Amit wrote in the other, related thread -
https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.cloudfoundry.org/thread/RYMBWNUMMWSDPIE3DZ5JUVN2VNYXDWU6/


This will not work with dynamic networks. Many jobs in cf-release rely
on data from BOSH to determine their IP so that configuration files can be
rendered up-front by the director rather than at runtime, requiring system
calls to determine IP. metron_agent is one such job, and it tends to be
colocated with each other job (it is what allows all system component logs
to be aggregated through the loggregator system), so this would require all
Cloud Foundry VMs to be on a manual network. You don't need to manually
pick the IPs, you just need to tell BOSH which IPs in the network not to
use and specify these in the "reserved" range.

Since so many different components depend on being able to determine
their IP via BOSH data, there's no quick workaround if you want to stick to
using dynamic networks, but we're aware of this current limitation.

Thanks,
Rob & Alvaro
CF Release Integration
Pivotal


Re: Request for Multibuildpack Use Cases

John Feminella <jxf@...>
 

Multibuildpack is absolutely useful and I'm excited for this proposal.

I encounter a lot of use cases for this. The most common is that an
application wants to pull in private dependencies during a future
dependency-resolution step of a later buildpack, but the dependency
resolver needs to be primed in some specific way. If you wait until
buildpack time it's too late.

On Heroku, for example, this is accomplished by having something like the
netrc buildpack (https://github.com/timshadel/heroku-buildpack-github-netrc),
adding a GITHUB_TOKEN environment variable, and then running your "real"
buildpack. The netrc BP runs first, allowing Bundler to see the private
dependencies.

best,
~ jf

On Tue, Apr 12, 2016 at 12:36 PM Jack Cai <greensight(a)gmail.com> wrote:

It would be more useful if the multi-buildpack can reference an admin
buildpack in addition to a remote git-hosted buildpack. :-)

Jack


On Tue, Apr 12, 2016 at 6:38 AM, David Illsley <davidillsley(a)gmail.com>
wrote:

In the past we've used the multi-buildpack to be able to use ruby sass to
compile SCSS for non-ruby projects (node and Java). In that case we used
the multi-buildpack and a .buildpacks file which worked reasonably well
(and was very clear).

On Mon, Apr 11, 2016 at 1:15 AM, Danny Rosen <drosen(a)pivotal.io> wrote:

Hi there,

The CF Buildpacks team is considering taking on a line of work to
provide more formal support for multibuildpacks. Before we start, we would
be interested in learning if any community users have compelling use cases
they could share with us.

For more information on multibuildpacks, see Heroku's documentation [1]

[1] -
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app


Re: dynamic networks

CF Runtime
 

As Amit wrote in the other, related thread - https://lists.cloudfoundry.org/archives/list/cf-dev(a)lists.cloudfoundry.org/thread/RYMBWNUMMWSDPIE3DZ5JUVN2VNYXDWU6/


This will not work with dynamic networks. Many jobs in cf-release rely on data from BOSH to determine their IP so that configuration files can be rendered up-front by the director rather than at runtime, requiring system calls to determine IP. metron_agent is one such job, and it tends to be colocated with each other job (it is what allows all system component logs to be aggregated through the loggregator system), so this would require all Cloud Foundry VMs to be on a manual network. You don't need to manually pick the IPs, you just need to tell BOSH which IPs in the network not to use and specify these in the "reserved" range.
Since so many different components depend on being able to determine their IP via BOSH data, there's no quick workaround if you want to stick to using dynamic networks, but we're aware of this current limitation.
Thanks,
Rob & Alvaro
CF Release Integration
Pivotal


Re: CF212 - golang app crash - how to perform the troubleshooting?

JT Archie <jarchie@...>
 

They are passed via environment variables and then your start command is
used to start the app.

This is an example of the startup script
<https://github.com/cloudfoundry/dea_ng/blob/master/lib/dea/starting/startup_script_generator.rb>
generated.

Does this help?


On Tue, Apr 12, 2016 at 7:52 AM, Rafal Radecki <radecki.rafal(a)gmail.com>
wrote:

Hi :)

I am starting an app and it crashes. In the meantime when it is trying to
start I login to appropriate warden container and after the app crashes I
try to start the go binary manually. The problem is that I am not able to
see in the environment variables like VCAP_SERVICES and I cannot pass its
content correctly to my app.

Can you tell me how to start my app inside the warden container once I
login through wsh with proper environment setup? I tried below:

root(a)19g7korv77m:/app# export
VCAP_SERVICES='{"mongodb26":[{"credentials":{"dbname":"...}
root(a)19g7korv77m:/app# bin/my_go_binary

but it is obviously not the proper way. How does cloudfoundry pass
variables to the process started in the warden container?

BR,
Rafal.


Re: Request for Multibuildpack Use Cases

Jack Cai
 

It would be more useful if the multi-buildpack can reference an admin
buildpack in addition to a remote git-hosted buildpack. :-)

Jack


On Tue, Apr 12, 2016 at 6:38 AM, David Illsley <davidillsley(a)gmail.com>
wrote:

In the past we've used the multi-buildpack to be able to use ruby sass to
compile SCSS for non-ruby projects (node and Java). In that case we used
the multi-buildpack and a .buildpacks file which worked reasonably well
(and was very clear).

On Mon, Apr 11, 2016 at 1:15 AM, Danny Rosen <drosen(a)pivotal.io> wrote:

Hi there,

The CF Buildpacks team is considering taking on a line of work to provide
more formal support for multibuildpacks. Before we start, we would be
interested in learning if any community users have compelling use cases
they could share with us.

For more information on multibuildpacks, see Heroku's documentation [1]

[1] -
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app


Re: Doppler/Firehose - Multiline Log Entry

Jim CF Campbell
 

Mike,

When you get a bit more desperate ;-) here is a nozzle plug in
<https://github.com/jtuchscherer/nozzle-plugin> for the CLI. It's attaches
to the firehose to display everything, but would be easy to modify to just
look at a single app, and sub out the magic token for newlines.

Jim

On Tue, Apr 12, 2016 at 9:56 AM, Mike Youngstrom <youngm(a)gmail.com> wrote:

Hi David,

The problem for me is that I'm searching for a solution that can works for
development (though less of a priority cause you can switch config between
dev and cf) and for viewing logs via "cf logs" in addition to a log
aggregator. I had hoped that /u2028 would work for viewing logs via "cf
logs" but it doesn't in bash. I'd need to write a plugin or something for
cf logs and train all my users to use it. Certainly possible but I'm not
that desperate yet. :)

Mike

On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com> wrote:

FWIW, the technique is to have your logging solution (eg, logback, log4j)
log a token (eg, \u2028) other than \n to denote line breaks in
your stack traces; and then have your log aggregation software replace that
token with a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else; eg
NEWLINE

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com> wrote:

Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution.
Hope you make progress on a metric format available to an app in a
container. I too would like a tracker link to such a feature if there is
one.

Thanks,
Mike

On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi Jim,

So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.

Mike

On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <jcampbell(a)pivotal.io
wrote:
Hi Mike and Alex,

Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for Logstash.
Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana
will display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

"]
^^^ Seems that passing a string with an actual newline in it is the
only way to make gsub work

}

to replace the token with a regular newline again so it displays
"properly" in Kibana.

[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>

[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>


On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?

Mike

On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963
--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


AWS / CF v233 deployment

Sylvain Gibier
 

Hi,

Trying to a fresh new installation of CF on AWS - I'm hitting the following
issue when the deployment is trying to update the api_z1/0 jobs.
I'm following the instruction -
http://docs.cloudfoundry.org/deploying/aws/cf-stub.html and as well
http://docs.cloudfoundry.org/deploying/common/consul-security.html to
generate the consul certificates.

According to the log - the consult_agent process failed to start.

{"timestamp":"1460475448.173879385","source":"confab","message":"confab.agent-client.verify-joined.members.request.failed","log_level":2,"data":{"error":"Get
http://127.0.0.1:8500/v1/agent/members: dial tcp 127.0.0.1:8

500: getsockopt: connection refused","wan":false}}

==> Starting Consul agent...

==> Error starting agent: Failed to start Consul client: Failed to load
cert/key pair: crypto/tls: failed to parse certificate PEM data
How can I debug this issue ?

Cheers,

sylvain


Re: Doppler/Firehose - Multiline Log Entry

Mike Youngstrom
 

Hi David,

The problem for me is that I'm searching for a solution that can works for
development (though less of a priority cause you can switch config between
dev and cf) and for viewing logs via "cf logs" in addition to a log
aggregator. I had hoped that /u2028 would work for viewing logs via "cf
logs" but it doesn't in bash. I'd need to write a plugin or something for
cf logs and train all my users to use it. Certainly possible but I'm not
that desperate yet. :)

Mike

On Tue, Apr 12, 2016 at 5:58 AM, David Laing <david(a)davidlaing.com> wrote:

FWIW, the technique is to have your logging solution (eg, logback, log4j)
log a token (eg, \u2028) other than \n to denote line breaks in
your stack traces; and then have your log aggregation software replace that
token with a \n again when processing the log messages.

If \u2028 doesn't work in your environment; use something else; eg NEWLINE

On Mon, 11 Apr 2016 at 21:12 Mike Youngstrom <youngm(a)gmail.com> wrote:

Finally got around to testing this. Preliminary testing show that "\u2028"
doesn't function as a new line character in bash and causes eclipse console
to wig out. I don't think "\u2028" is a viable long term solution.
Hope you make progress on a metric format available to an app in a
container. I too would like a tracker link to such a feature if there is
one.

Thanks,
Mike

On Mon, Mar 14, 2016 at 2:28 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

Hi Jim,

So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.

Mike

On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

Hi Mike and Alex,

Two things - for Java, we are working toward defining an enhanced
metric format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for Logstash.
Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana will
display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

"]
^^^ Seems that passing a string with an actual newline in it is the
only way to make gsub work

}

to replace the token with a regular newline again so it displays
"properly" in Kibana.

[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>

[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>


On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?

Mike

On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How
correctly deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros


--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963


Re: CPU weight of application

Eric Malm <emalm@...>
 

Hi, Sam,

No, the LRP generated for the CF app will have its CPU weight set
proportionally to its desired memory. Keep in mind that for an app running
on Linux, the CPU weight corresponds to the cpu.shares cgroup, which
enforces a relative amount of CPU usage amongst processes only if there is
contention over CPU. If nothing else is competing for CPU on the VM, your
process can use as much as it is capable of using.

Thanks,
Eric, CF Runtime Diego PM

On Sun, Apr 10, 2016 at 8:19 PM, Sam Dai <sam.dai(a)servicemax.com> wrote:

Hello,
According to code
https://github.com/cloudfoundry-incubator/nsync/blob/01a624d23cb683f35c88c4160205c4ad880faaf0/recipebuilder/recipe_builder.go#L72-L84
, diego apps scale the number of allocated cpu shares linearly with the
amount of memory when allocated memory is > 256MB and < 8192MB, is there
a way to allocate extra cpu to an app that happens to need less memory?

Thanks,
Sam

4821 - 4840 of 9408