Firehose vs Pivotal Ops metrics


Qing Gong
 

I set up the open source CF and got the metrics streamed out of the firehose. I also read document about Pivotal Ops Metrics in here.

http://docs.pivotal.io/pivotalcf/customizing/use-metrics.html

What are the relationship between the two sets of metrics? The Pivotal one uses JMX and has a completely different set of metrics. Why do we have metrics in two places?

Also, are there any documents about what metrics are streamed from the firehose? I could not find any document on this other than seeing the actual data streamed from the firehose.

Thanks!


Erik Jasiak
 

Hi Qing,

First, Ops Metrics is a closed-source addition to Pivotal Cloud Foundry.
The metrics exposed by Ops Metrics today come from the collector[1] and
tomorrow from both the collector and firehose. All other questions about
Ops Metrics should be directed to Pivotal support.

Metrics from the collector in open-source CF are available as OpenTSDB,
graphite, datadog, or CloudWatch formats.

The loggregator team has known for some time that CF would need to phase
out the collector. With the addition of the loggregator firehose, CF
components could broadcast metrics via another route. Several pre-Diego
components still rely on the collector today (and in many cases, now
also transmit metrics out the firehose as well). For maintaining
backward compatibility the loggregator team has been busily adding a
nozzle to feed firehose data back into the collector[2] . Overall, the
goal is that both metrics resources have parity where possible, but we
will phase out future metrics going to the collector over time.

Future CF components (Diego and after) will broadcast metrics along the
firehose only, and metrics made available in the format of choice via
nozzle.

Our metrics documentation is ongoing. Because metrics travel out the
loggregator or collector system - but originate from other CF components
- keeping track of what metrics are in the system is a challenge. We
have some basic public info we've gathered [3], and are working to wrap
this in proper documentation now.

Finally, we've been working to document all this, primarily in the
readmes of our repositories. It's been clear for some time that this is
not sufficient, and we're working on alternate ways to get the info out
into the community's hands, while also working to produce core CF
documentation.

Hope this helps; stay tuned as we get more info out there.
Erik Jasiak
PM - Loggregator

[1] https://github.com/cloudfoundry/collector
[2] https://github.com/cloudfoundry-incubator/varz-firehose-nozzle
[3]
https://docs.google.com/spreadsheets/d/176yIaJChXEmvm-CjybmwopdRGQfDGrSzo3J_Mx8mMnk


Qing Gong wrote:


I set up the open source CF and got the metrics streamed out of the
firehose. I also read document about Pivotal Ops Metrics in here.

http://docs.pivotal.io/pivotalcf/customizing/use-metrics.html

What are the relationship between the two sets of metrics? The Pivotal
one uses JMX and has a completely different set of metrics. Why do we
have metrics in two places?

Also, are there any documents about what metrics are streamed from the
firehose? I could not find any document on this other than seeing the
actual data streamed from the firehose.

Thanks!


Qing Gong
 

Erik,

That's very helpful, thank you!

Qing