Topics

Proposal to end-of-life the varz nozzle


Jim CF Campbell
 

The initial implementation of CF metrics used the Collector to poll /varz
endpoints of various components and then emit them to consumers such as
Datadog, Graphite and others. With the implementation of Loggregator and
the firehose, the default metric path became nozzles attached to the
firehose.

The varz nozzle was created to consume metrics from the firehose and then
present them via polled /varz endpoints to the Collector to allow backward
compatibility. This is a long-deprecated technique and the Loggregator team
is proposing to end support and maintenance of the varz nozzle.

This will be followed soon by an effort to end-of-life the Collector itself.

We would like community input on this.

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Marco Voelz
 

Dear Jim,

I understand that you want to get rid of a deprecated approach. However, I'm not sure if there is a parity for the new approach yet and if we've seen a grace period allowing people to switch.

Can you give us an up-to-date overview about which metrics are available over the firehose and which *only* over /varz? For example, for quite a long time the metrics of DEAs were not available in the firehose. I know that this has recently changed [1], but there hasn't been a cf-release include these changes.

As this is only one example I remember from the top of my head I'd like to

* be sure that we actually have everything available in the firehose that we're getting from /varz today
* after that is the case wait some time so people can switch and use the new approach

Thanks and warm regards
Marco

[1] https://github.com/cloudfoundry/dea_ng/commit/a1aa10a887c5facb10270f82bba6214529fcfa3d#diff-9edf5787563a04c03ce32d85b1f6d7ab

On 19/02/16 01:21, "Jim CF Campbell" <jcampbell(a)pivotal.io<mailto:jcampbell(a)pivotal.io>> wrote:

The initial implementation of CF metrics used the Collector to poll /varz endpoints of various components and then emit them to consumers such as Datadog, Graphite and others. With the implementation of Loggregator and the firehose, the default metric path became nozzles attached to the firehose.

The varz nozzle was created to consume metrics from the firehose and then present them via polled /varz endpoints to the Collector to allow backward compatibility. This is a long-deprecated technique and the Loggregator team is proposing to end support and maintenance of the varz nozzle.

This will be followed soon by an effort to end-of-life the Collector itself.

We would like community input on this.

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Jim CF Campbell
 

Hi Marco,

the varz nozzle is only for metrics that are *already* in the firehose. It
connects to the firehose and offers a /varz endpoint to the metrics that
come through the firehose. It is designed to support systems that only want
their metrics via varz. There are metrics from DEA and HM9000 that are
currently not in the firehose and only available via /varz.

Does that make sense?

On Fri, Feb 19, 2016 at 11:47 AM, Voelz, Marco <marco.voelz(a)sap.com> wrote:

Dear Jim,

I understand that you want to get rid of a deprecated approach. However,
I'm not sure if there is a parity for the new approach yet and if we've
seen a grace period allowing people to switch.

Can you give us an up-to-date overview about which metrics are available
over the firehose and which *only* over /varz? For example, for quite a
long time the metrics of DEAs were not available in the firehose. I know
that this has recently changed [1], but there hasn't been a cf-release
include these changes.

As this is only one example I remember from the top of my head I'd like to

- be sure that we actually have everything available in the firehose
that we're getting from /varz today
- after that is the case wait some time so people can switch and use
the new approach

Thanks and warm regards
Marco

[1]
https://github.com/cloudfoundry/dea_ng/commit/a1aa10a887c5facb10270f82bba6214529fcfa3d#diff-9edf5787563a04c03ce32d85b1f6d7ab

On 19/02/16 01:21, "Jim CF Campbell" <jcampbell(a)pivotal.io> wrote:

The initial implementation of CF metrics used the Collector to poll /varz
endpoints of various components and then emit them to consumers such as
Datadog, Graphite and others. With the implementation of Loggregator and
the firehose, the default metric path became nozzles attached to the
firehose.

The varz nozzle was created to consume metrics from the firehose and then
present them via polled /varz endpoints to the Collector to allow backward
compatibility. This is a long-deprecated technique and the Loggregator team
is proposing to end support and maintenance of the varz nozzle.

This will be followed soon by an effort to end-of-life the Collector
itself.

We would like community input on this.

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963



--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Marco Voelz
 

Dear Jim,

thanks for your answer and the detailed explanation of the /varz nozzle. I was more worried about the latter part of your first email, in which you say you want to EOL the collector "soon". For that to happen, I'd like to make sure that we don't lose any metrics we currently rely on.

In terms of removing the /varz nozzle: I'd probably leave that in for as long as the collector is there. Forcing people to get metrics from two different sources to get the full picture is a possible workaround, but not the nicest solution here.

Warm regards
Marco

On 19/02/16 23:50, "Jim CF Campbell" <jcampbell(a)pivotal.io<mailto:jcampbell(a)pivotal.io>> wrote:

Hi Marco,

the varz nozzle is only for metrics that are *already* in the firehose. It connects to the firehose and offers a /varz endpoint to the metrics that come through the firehose. It is designed to support systems that only want their metrics via varz. There are metrics from DEA and HM9000 that are currently not in the firehose and only available via /varz.

Does that make sense?

On Fri, Feb 19, 2016 at 11:47 AM, Voelz, Marco <marco.voelz(a)sap.com<mailto:marco.voelz(a)sap.com>> wrote:
Dear Jim,

I understand that you want to get rid of a deprecated approach. However, I'm not sure if there is a parity for the new approach yet and if we've seen a grace period allowing people to switch.

Can you give us an up-to-date overview about which metrics are available over the firehose and which *only* over /varz? For example, for quite a long time the metrics of DEAs were not available in the firehose. I know that this has recently changed [1], but there hasn't been a cf-release include these changes.

As this is only one example I remember from the top of my head I'd like to

* be sure that we actually have everything available in the firehose that we're getting from /varz today
* after that is the case wait some time so people can switch and use the new approach

Thanks and warm regards
Marco

[1] https://github.com/cloudfoundry/dea_ng/commit/a1aa10a887c5facb10270f82bba6214529fcfa3d#diff-9edf5787563a04c03ce32d85b1f6d7ab

On 19/02/16 01:21, "Jim CF Campbell" <jcampbell(a)pivotal.io<mailto:jcampbell(a)pivotal.io>> wrote:

The initial implementation of CF metrics used the Collector to poll /varz endpoints of various components and then emit them to consumers such as Datadog, Graphite and others. With the implementation of Loggregator and the firehose, the default metric path became nozzles attached to the firehose.

The varz nozzle was created to consume metrics from the firehose and then present them via polled /varz endpoints to the Collector to allow backward compatibility. This is a long-deprecated technique and the Loggregator team is proposing to end support and maintenance of the varz nozzle.

This will be followed soon by an effort to end-of-life the Collector itself.

We would like community input on this.

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963<tel:303.618.0963>





--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963


Jim CF Campbell
 

Hi Marco,

All metrics are now available from the firehose. The only metrics
continuing to flow through the Collector are from DEA, which are also
flowing through the firehose.

The collector currently supports emitting metrics to
Datadog
- Graphite
- OpenTSDB
- AWS Cloud Watch

All are supported via nozzles except Cloud Watch.

So except for Cloud Watch, everyone can continue working in their own way
with the Nozzle instead of the Collector. Cloud Watch people can continue
to use Collector, we just won't maintain it. It has not had a commit since
Sept 2015. We'd be happy to advise someone who wanted to build a Cloud
Watch nozzle.

Jim

On Mon, Feb 22, 2016 at 3:01 AM, Voelz, Marco <marco.voelz(a)sap.com> wrote:

Dear Jim,

thanks for your answer and the detailed explanation of the /varz nozzle. I
was more worried about the latter part of your first email, in which you
say you want to EOL the collector "soon". For that to happen, I'd like to
make sure that we don't lose any metrics we currently rely on.

In terms of removing the /varz nozzle: I'd probably leave that in for as
long as the collector is there. Forcing people to get metrics from two
different sources to get the full picture is a possible workaround, but not
the nicest solution here.

Warm regards
Marco

On 19/02/16 23:50, "Jim CF Campbell" <jcampbell(a)pivotal.io> wrote:

Hi Marco,

the varz nozzle is only for metrics that are *already* in the firehose. It
connects to the firehose and offers a /varz endpoint to the metrics that
come through the firehose. It is designed to support systems that only want
their metrics via varz. There are metrics from DEA and HM9000 that are
currently not in the firehose and only available via /varz.

Does that make sense?

On Fri, Feb 19, 2016 at 11:47 AM, Voelz, Marco <marco.voelz(a)sap.com>
wrote:

Dear Jim,

I understand that you want to get rid of a deprecated approach. However,
I'm not sure if there is a parity for the new approach yet and if we've
seen a grace period allowing people to switch.

Can you give us an up-to-date overview about which metrics are available
over the firehose and which *only* over /varz? For example, for quite a
long time the metrics of DEAs were not available in the firehose. I know
that this has recently changed [1], but there hasn't been a cf-release
include these changes.

As this is only one example I remember from the top of my head I'd like to

- be sure that we actually have everything available in the firehose
that we're getting from /varz today
- after that is the case wait some time so people can switch and use
the new approach

Thanks and warm regards
Marco

[1]
https://github.com/cloudfoundry/dea_ng/commit/a1aa10a887c5facb10270f82bba6214529fcfa3d#diff-9edf5787563a04c03ce32d85b1f6d7ab

On 19/02/16 01:21, "Jim CF Campbell" <jcampbell(a)pivotal.io> wrote:

The initial implementation of CF metrics used the Collector to poll /varz
endpoints of various components and then emit them to consumers such as
Datadog, Graphite and others. With the implementation of Loggregator and
the firehose, the default metric path became nozzles attached to the
firehose.

The varz nozzle was created to consume metrics from the firehose and then
present them via polled /varz endpoints to the Collector to allow backward
compatibility. This is a long-deprecated technique and the Loggregator team
is proposing to end support and maintenance of the varz nozzle.

This will be followed soon by an effort to end-of-life the Collector
itself.

We would like community input on this.

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963




--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963



--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963