Thank you all for the feedback. This is very helpful. I understand your concern about the short timeframe. I am very open to extending the timeframe to allow everybody to move of the firehose in a non-stressful way, although I hope that we don't have to extend it to 12 months. 

What help would most to migrate away from  the firehose faster? So far in this thread, I saw a few open source firehose integrations mentioned:

1. firehose-exporter (for prometheus)

2. firehose-to-syslog (for logsearch) 

3. cf-java-client

If we (the loggregator-team) helped migrating these integrations to the log-stream endpoint (from the RLP), would that help? If not, are there other things we could do to improve the migration process?

Thanks again for the feedback. We really appreciate it and are listening.

Johannes

On Thu, Feb 14, 2019 at 12:19 PM Mike Youngstrom <youngm@...> wrote:

We have developed some custom monitoring using the firehose and Splunk based on the cf-java-client firehose support.  The jmxconsumer has some known memory leaks and would prefer to wait for official cf-java-client support (https://github.com/cloudfoundry/cf-java-client/issues/904).  The sooner cf-java-client has 2.x support the sooner we'll be off.  As it stands 3 months is a little quick for us as well for that reason.

Thanks,

Mike

On Thu, Feb 14, 2019 at 11:58 AM Jon Price <jon.price@...> wrote:

Same concerns for us, we too run the open source CF deployment along with the prometheus-boshrelease and logsearch-for-cloudfoundry release.

 

Jon Price

Intel Corp.

 

From: cf-dev@... <cf-dev@...> On Behalf Of via Lists.Cloudfoundry.Org
Sent: Thursday, February 14, 2019 9:10 AM
To: cf-dev@...
Subject: Re: [cf-dev] [Proposal] Deprecation of the firehose endpoint

 

Hi Johannes,

 

I work on GOV.UK PaaS - we run an open source CF deployment. Our deployment also uses the firehose exporter that Neil mentions (via prometheus-boshrelease).

 

We probably don't have enough people to help much with migrating these things, but they form a core part of our platform monitoring, so it would be a problem for us if they stopped working.

 

I agree with Neil and Simon that three months may not be enough time for the community to adopt the new things.

 

Thanks,

Richard

 

 

 

On Thu, 14 Feb 2019 at 15:22, Simon D Moser <smoser@...> wrote:

Johannes,

just echoing what Neil is saying below: We also feel that three months is probably too short of a runway for this. Let's discuss the details, but 3 months is likely not good enough for us, too.



From:        "Neil MacDougall" <neil.macdougall@...>
To:        cf-dev@...
Date:        12/02/2019 18:05
Subject:        Re: [cf-dev] [Proposal] Deprecation of the firehose endpoint
Sent by:        cf-dev@...

---

Johannes,

Thank your for the notification.

We use the firehose in Stratos to stream logs for applications and the firehose itself for users/admins to view in the UI.

We also use the CF Firehose exporter (https://github.com/bosh-prometheus/firehose_exporter) as does https://github.com/bosh-prometheus/prometheus-boshrelease) to retrieve metrics from the firehose and store those in Prometheus for display within Stratos. Do you know what the plan is for these to be updated?

It is a large amount of work for us to make the changes for the removal of the firehose and I think 3 months is too aggressive a time frame for us to be able to complete those - this is not something we’ve factored into our planning, so I think we’d need a 12 month window on the firehose removal to be safe.

We have users using Stratos on a variety of Cloud Foundry distributions, which we won’t be able to guarantee will be updated in your proposed three month window, so we will most likely have to make updates to allow Stratos to work with systems that use the current firehose and those that use the newer APIs.

I’m happy to discuss this further with you directly if that would help.

Regards,

Neil


On 11 Feb 2019, at 23:54, Johannes Tuchscherer <jtuchscherer@...> wrote:

Hi there,

Following the last thread about the deprecation of the /containermetrics endpoint, the Loggregator team would like to continue on its path of deprecations. Next on the chopping block is the /firehose endpoint. We understand that this endpoint is used by a few integrations, so we will provide reference implementations and documents for migrating away from the firehose to the newer loggregator endpoints (namely, the RLP Gateway and LogCache). Depending on the feedback we receive to this email, we would like to proceed with the removal of the firehose endpoint in three months - meaning that the loggregator release being cut in three month’s time won't have support for that endpoint anymore.

You can find some example consumers and the documentation of the RLP here:
Go: https://github.com/cloudfoundry/log-stream-cli
Go Client: https://github.com/cloudfoundry/go-loggregator/blob/master/rlp_gateway_client.go
Java: https://github.com/cloudfoundry-community/jmx-consumer-release/tree/develop/src/jmxconsumer
Docs: https://github.com/cloudfoundry/loggregator/blob/master/docs/rlp_gateway.md

Here are some example consumers and the documentation of the Log Cache:
Go: https://github.com/cloudfoundry/log-cache-cli
Go Client: https://github.com/cloudfoundry/log-cache/tree/master/pkg/client
Docs: https://github.com/cloudfoundry/log-cache/blob/master/README.md


Our desire to remove endpoints on the trafficcontroller is based on our plan to get rid of the trafficcontroller - and then everything in loggregator that still references the V1 api and protocol. This will help us to significantly reduce code complexity, but it will also allow us to operate much more efficiently. This should result in reduced overhead for the logging pipeline and ultimately lower infrastructure costs for the logging components in a CF deployment.

Please let us know if you have any comments or concerns.

Johannes
PM of #loggregator