Re: Logstash and Multiline Log Entry
|
Erik Jasiak
Hi Steve and Simon; hello again Mike,
toggle quoted message
Show quoted text
First, apologies for the delay in reply on this one- I've also been trying to come up with a simple, short answer to this problem. I failed. Here are the high-level, non-technical answers: 1) Yes, we'd love to enable multi-line logging. Regardless of any other challenges, we know that there's interest. 2) The problem is multi-layered, and extends beyond loggregator. 2a) Most of the problems with multi-line logging that overlap loggregator also overlap "general scalability" - problems we've been handling as part of moving toward collector retirement. 3) We have a hack day project looking at anything "quick and dirty" to help fix this. 4) Redirecting app logs have known workarounds (eg in Java: via log4j or similar) while we tackled this - not preferred at all, but do-able. ######### Technical answers: Loggregator's goals are "Fast, thorough, dumb." Multi-line logging - as handled by loggregator - has no clean way of working at the moment w/o violating "fast" or "dumb" principles today. We're getting there though. Here's how we've been working towards a fix: * Syslog drains were not performant enough, or could not handle large java traces - something we recently fixed[1][2][3] and are going to email about separately. * Horizontal scalability allows for overall better performance and reliability, but pushes the cost on data consistency to the edges of loggregator (hence nozzles, injectors.) * Loggregator's dropsonde protocol didn't allow for a clean way to enforce/tag multi-line data consistency - something we are about to put forward a proposal to remedy. ** Timestamps are not a clean mechanism for reliably re-assembling a multi-line log - some combination of app-instance and order-of-output would need to be tacked on, or a decent vector-time implementation. We'd need a way to add this metadata that would allow for re-assembly (see protocol item above). We'd also have to add extra info at DEA or garden without sacrificing performance - and we know that the DEA logging agent today already has questions around "acceptable" performance. So a multiline fix intersects our goals today. I will do my best to highlight stories that help us with multi-line logging, and we need to do a better job at communicating that we're working toward it, even if it's not the obvious target goal. , Erik [1] https://www.pivotaltracker.com/story/show/99494586 [2] https://www.pivotaltracker.com/story/show/97928938 [3] https://www.pivotaltracker.com/story/show/100163298 Steve Wall wrote: Now I see what that means. Each line of a multiline log message could |
|
|