1. Messages
  2. [Proposal] Deprecation of the firehose endpoint

Re: [Proposal] Deprecation of the firehose endpoint

Johannes Tuchscherer
 

Hello everybody,

after a lot of deliberation and many good conversations, we came to the conclusion that we will go ahead and remove the firehose endpoint in six months. So, in the first release published after September 14th, there will be no more firehose endpoint. Until then, we will continue to help the community to migrate commonly used firehose integrations over to using the new endpoints located on the Reverse Log Proxy.

Just to reiterate the benefit of this effort, this is an important step to lower overhead and complexity for cloud foundry operator. Also, it will bring more adoption to the new endpoints which should be easier to consume and put less stress on the logging system.

Thanks again for everybody who reached out to us either on this list or on Slack or a separate email thread. Your input was very valuable.

Please let me know if you have any questions or concerns,
Johannes 

On Tue, Mar 5, 2019 at 11:52 AM Johannes Tuchscherer <jtuchscherer@...> wrote:
Hi Neil,

Thanks for the feedback. We actually created PRs for both, firehose-exporter[0] and to firehose-to-syslog[1]. We will continue working with the maintainers to get these pulled in. 

Best,
Johannes


On Tue, Mar 5, 2019 at 11:26 AM Neil MacDougall <neil.macdougall@...> wrote:
Johannes,

A 6 month deprecation window works for Stratos - that should be fine for us to update.

If you can work with the firehose-exporter project to get that updated and off the firehose within the next 2 or 3 months, that would be super helpful and give us time to integrate and test, assuming its a drop-in replacement for the current firehose based version.

Many thanks,

Neil

On 25 Feb 2019, at 23:51, Johannes Tuchscherer <jtuchscherer@...> wrote:

Hi everybody,

thanks for the continuous stream of feedback. We are definitely going to support the firehose-exporter and firehose-to-syslog projects off the firehose to the RLP. The stories in our backlog are coming up soon. Stanislav, do you think that looking at what we did with the firehose-to-syslog project would help you to migrate the kafka-firehose-nozzle?

Also, based on the feedback I have received so far I am open to increasing the deprecation window to 6 months. Would that be acceptable?

Thanks,
Johannes

On Sun, Feb 24, 2019 at 4:22 PM Stanislav German-Evtushenko <s.germanevtushenko@...> wrote:
Hi Johannes,

We are heavy users of loggregator and we have the same concerns as others in the thread.

We use loggregator for:
- storing all application logs for 6 month (regulations requirement) using kafka-firehose-nozzle
- getting all platform metrics for monitoring purposes using kafka-firehose-nozzle
- getting all applications metrics to provide monitoring dashboard using kafka-firehose-nozzle and cf-metrics-refinery (see https://cloudfoundry.slack.com/messages/C6WFZ5K0F/p1531268612000101)

Another concern is autoscaler (https://github.com/cloudfoundry-incubator/app-autoscaler-release). It is relying on loggregator and I haven't seen any updates on moving out of it yet.

Best regards,
Stanislav



Join {cf-dev@lists.cloudfoundry.org to automatically receive all group messages.