I don't believe there is a solution in raw Loggregator to fix this.
Multi line log messages is a major issue we deal with regularly today in
our deployment. Although there are workarounds to the issue like the
logstash config you posted, it is somewhat dependent upon how the
application logs are formatted which is sub optimal.
I'd personally like to see a syslog port available to the app container
that applications can send logs to in addition to channeling STDERR and
STDOUT. That would make issues like much easier to handle downstream and
would help eliminate deployed app specific hacks like we have to do today
in logstash.
I talked with a LAMB engineer at Summit and they didn't believe syslog for
deployed apps was anywhere on the LAMB roadmap. Can anyone confirm that?
I could have sworn I'd heard mention of syslog for app mentioned in the
past.
Mike
On Mon, May 18, 2015 at 9:32 AM, Li, George <guangxing.li(a)pearson.com>
wrote:
James,
so basically Loggregator does not support such correlation itself and it
is totally left to the downstream processor to do the aggregation, right?
When my service write exception stack trace into the log as follows:
2015-05-18 14:40:41 +0000 WARN
com.pearson.grid.registrar.commonutils.logging.LoggingMessageObserver
[LoggingMessageObserver.java:125] GET http://localhost:3004/ threw
exception: FileNotFoundException -
E:\Src\Registrar\escrow\service\config\version.properties (The system
cannot find the file specified)
java.io.FileInputStream.open(Native Method)
java.io.FileInputStream.<init>(FileInputStream.java:138)
com.pearson.grid.registrar.commonutils.Version.readProperties(Version.java:77)
Loggregator sends the following to downstream logstash:
2015-05-18 14:40:41 +0000 [App/0] OUT [escrow] [pool-3-thread-1052] WARN
com.pearson.grid.registrar.commonutils.logging.LoggingMessageObserver
[LoggingMessageObserver.java:142] GET http://escrow.cf.pd-cloud.com/
threw exception: FileNotFoundException - config\version.properties (No such
file or directory)
2015-05-18 14:40:41 +0000 [App/0] OUT java.io.FileInputStream.open(Native
Method)
2015-05-18 14:40:41 +0000 [App/0] OUT
java.io.FileInputStream.<init>(FileInputStream.java:146)
2015-05-18 14:40:41 +0000 [App/0] OUT
com.pearson.grid.registrar.commonutils.Version.readProperties(Version.java:77)
Ideally we want to keep all these as a single log "event" in the final
log. The way to do this in logstash is to config logstash so it would
aggregate all lines not staring with a timestamp (or any identifiable
marker) to previous lines:
input {
file {
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
...
}
}
So I think my exact question is if there is a similar way to config
loggregator so it does the same thing.
In the previous thread
https://groups.google.com/a/cloudfoundry.org/forum/?utm_medium=email&utm_source=footer#!msg/vcap-dev/B1W6_vO0oyo/84X1eAtFsKoJ,
David Lee said "...As for multiline output, we should have fixed this a
couple of builds back...", I wonder what was really fixed related to
multiline output at that time.
Thanks.
George.
On Sun, May 17, 2015 at 8:26 AM, James Bayer <jbayer(a)pivotal.io> wrote:
george,
can you explain the scenario that is impacting you currently? e.g.
reading a java stack trace? the app index is shown in the log line. when
using the loggregator websocket client to retrieve logs you should also get
the application id. perhaps that is enough to correlate whatever UI you
have on top of logstash?
On Tue, May 12, 2015 at 7:40 AM, Li, George <guangxing.li(a)pearson.com>
wrote:
Hi,
this is a follow up on the archived posting
https://groups.google.com/a/cloudfoundry.org/forum/?utm_medium=email&utm_source=footer#!msg/vcap-dev/B1W6_vO0oyo/84X1eAtFsKoJ.
I cannot find any new postings on that thread.
I am using Cloud Foundry version
"6.11.2-2a26d55-2015-04-27T21:11:44+00:00" and want to know what options I
have to handle multiple line logs in a multi-tenant environment. Since
multiple instances of multiple applications are all sending logs to a
single Logstash server, is it best to avoid having multiple lines in my
log? I can live with sticking to single line logs except for outputting
exception stack trace, not to mention that I only have control over my
code.
Thanks.
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
--
Thank you,
James Bayer
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
