Recently the Loggregator team has been researching the implications of streaming and the lack of ordering guarantee associated with it. I wanted to post a few findings as it is a common inquiry and something the community should be aware of best development practices around.
- If you are developing a client that displays a stream of logs to users you should consider ordering them to improve the debugging UX. There are a couple common techniques for this. One is to batch the logs and display whatever logs you have in that timeframe sorted by timestamp. This is good for CLI's. For web clients you can use dynamic HTML to insert older logs into the sorting as they appear. This is nice because by the time a user grabs the content for a copy paste it will likely be both complete and in order.
- The CLI batches and sorts logs for the user, but this functionality was taken out in a recent version of the CLI and only recently re-introduced in version 6.33.1. The cf cli uses a wait period of 300ms for batching the logs which is not noticeable in my experience.
- The firehose does not offer a deterministic routing mechanism for ordering.
- Syslog drains are serviced by two adapters for HA reasons so ingestors may need to be configured to take full advantage of the nanosecond precision on timestamps and ensure proper sorting at rest. Here are some helpful instructions we have found for both ELK and Splunk