Planned support for logs & metrics of service instances in loggregator?
Dear friends of logs and metrics
At the last CF summit in Basel, SAP’s service fabric team and our team (Swisscom) had discussions about a common use case we face:
Our devs are very happy how they can access logs of their apps using cf logs or the syslog drain.
However, they do currently have no standard way to access their service instances’ logs and metrics (think cf service-logs or a syslog drain for service instances).
This is especially critical for SAP and Swisscom as service providers who produce managed services, but we assume other CF providers will also face the same use case.
Loggregator seems to be a natural fit for such extensions, however, there are certain concerns:
From an earlier quick chat with Alex (Loggregator PM) I recall that widening the scope of loggregator was problematic (especially considered the added load this would bring).
Also, we think having this decoupled from CF (application runtime) could be beneficial, so that we could reuse it for the CF container runtime and possibly other service consumers.
Are there any efforts plan to extend Loggregator in this direction or open it up for extension?
Hi Matthias -
We have this vision as well and we recently just got together to discuss related topics as a new initiative and there has been some chatter about it in the Loggregator github repo. Historically there has been some concern about the load increase on loggregator with platform logs being included and we are still sensitive to this concern but we feel confident that recent loggregator versions can be scaled to handle the load.
Additionally we just released a new feature in cf-deployment in 1.9 I haven't had a chance to tell the community about. App Developers can now send app metrics through syslog drains. You'll need to download a plugin but this "metric-drain" capability is now available in the latest cf-deployment. We'd appreciate hearing any feedback you have on it.
We are also going to run an experiment with the team that is currently supporting the syslog-release, to send platform logs into Loggregator. This will likely remain experimental for some time but I imagine it could be easily opted into using an opsfile. I have some concerns about the scalability of this feature as your platform grows so this option becoming first class will require a bit of data from the experimentation. It also poses some migration challenges for those using a log consumption strategy like firehose to syslog.
Finally we are taking over a repo called the service metrics release that many Pivotal developed services utilize for sending metrics into Loggregator. Once we are able to appropriately tag metrics with service instance details like org and space, Loggregator is capable of routing those metrics through the aforementioned "metric drain". This requires some additional work in Loggregator as well but the end result we are shooting for is something we call a "space drain" which would allow an app developer to drain everything from their space; both apps and service instances. I framed this as application runtime specific, but we'd like to keep the ideas for the routing generic at the envelope level so that we could re-use the same concepts of draining for pods in container runtime.