Re: Planned support for logs & metrics of service instances in loggregator?
Hi Matthias -
We have this vision as well and we recently just got together to discuss related topics as a new initiative and there has been some chatter about it in the Loggregator github repo. Historically there has been some concern about the load increase on loggregator with platform logs being included and we are still sensitive to this concern but we feel confident that recent loggregator versions can be scaled to handle the load.
Additionally we just released a new feature in cf-deployment in 1.9 I haven't had a chance to tell the community about. App Developers can now send app metrics through syslog drains. You'll need to download a plugin but this "metric-drain" capability is now available in the latest cf-deployment. We'd appreciate hearing any feedback you have on it.
We are also going to run an experiment with the team that is currently supporting the syslog-release, to send platform logs into Loggregator. This will likely remain experimental for some time but I imagine it could be easily opted into using an opsfile. I have some concerns about the scalability of this feature as your platform grows so this option becoming first class will require a bit of data from the experimentation. It also poses some migration challenges for those using a log consumption strategy like firehose to syslog.
Finally we are taking over a repo called the service metrics release that many Pivotal developed services utilize for sending metrics into Loggregator. Once we are able to appropriately tag metrics with service instance details like org and space, Loggregator is capable of routing those metrics through the aforementioned "metric drain". This requires some additional work in Loggregator as well but the end result we are shooting for is something we call a "space drain" which would allow an app developer to drain everything from their space; both apps and service instances. I framed this as application runtime specific, but we'd like to keep the ideas for the routing generic at the envelope level so that we could re-use the same concepts of draining for pods in container runtime.