Re: Logstash and Multiline Log Entry
Hi Steve and Simon; hello again Mike,toggle quoted message Show quoted text
First, apologies for the delay in reply on this one- I've also
been trying to come up with a simple, short answer to this problem. I
Here are the high-level, non-technical answers:
1) Yes, we'd love to enable multi-line logging. Regardless of any
other challenges, we know that there's interest.
2) The problem is multi-layered, and extends beyond loggregator.
2a) Most of the problems with multi-line logging that overlap
loggregator also overlap "general scalability" - problems we've been
handling as part of moving toward collector retirement.
3) We have a hack day project looking at anything "quick and dirty"
to help fix this.
4) Redirecting app logs have known workarounds (eg in Java: via
log4j or similar) while we tackled this - not preferred at all, but do-able.
Technical answers: Loggregator's goals are "Fast, thorough, dumb."
Multi-line logging - as handled by loggregator - has no clean way of
working at the moment w/o violating "fast" or "dumb" principles today.
We're getting there though.
Here's how we've been working towards a fix:
* Syslog drains were not performant enough, or could not handle
large java traces - something we recently fixed and are going
to email about separately.
* Horizontal scalability allows for overall better performance
and reliability, but pushes the cost on data consistency to the edges of
loggregator (hence nozzles, injectors.)
* Loggregator's dropsonde protocol didn't allow for a clean way
to enforce/tag multi-line data consistency - something we are about to
put forward a proposal to remedy.
** Timestamps are not a clean mechanism for reliably
re-assembling a multi-line log - some combination of app-instance and
order-of-output would need to be tacked on, or a decent vector-time
implementation. We'd need a way to add this metadata that would allow
for re-assembly (see protocol item above). We'd also have to add extra
info at DEA or garden without sacrificing performance - and we know that
the DEA logging agent today already has questions around "acceptable"
So a multiline fix intersects our goals today. I will do my best
to highlight stories that help us with multi-line logging, and we need
to do a better job at communicating that we're working toward it, even
if it's not the obvious target goal.
Steve Wall wrote:
Now I see what that means. Each line of a multiline log message could