Re: [abacus-perf] Persisting Metrics performance
One of the thing that I want to do is to persist the metrics performancecollected by abacus-perf.
Interesting feature! It'd be good to understand what you're trying to do
with that data (I think Assk for asking a similar question) as that'll help
us provide better implementation suggestions.
The first solution is to use Turbine......
The problem with this is that Turbine persists the metrics datawhen there are no activity in the application, and it will flush the metrics
data when a new stats come in.
Not sure I'm following here. Can you give a bit more details to help us
understand how Turbine alters the stats and what problems that causes in
your collection + store logic?
Another solution is to mimic what the hystrix module is doing: Instead of streamingthe metrics to the hystrix dashboard, I would post to the database.
The Abacus-hystrix module responds to GET /hystrix.stream requests, and
doesn't do anything unless a monitor requests the stats. I'm not sure that
pro-actively posting the stats to a DB from each app instance will work so
well... as IMO that'll generate a lot of DB traffic from all these app
instances, will slow down these apps, and won't give you an aggregated
stats at the app level anyway (more on that below, however).
Here's a few more suggestions for you:
a) Give us a bit more context on how you intend to use the data you're
storing... if this is for use with Graphite for example, there's already a
number of blog posts out there that cover that; if you'd like to store the
data in ELK for searching then you might want to log these metrics and flow
them to ELK as part of your logs; if you'd like to store the data in your
own DB and render it using custom made dashboards later then we can explore
b) Try to leverage the current flow (with app instances providing stats on
demand at a /hystrix.stream endpoint and an external monitoring app
collecting these stats) rather than creating yet another completely
different flow; looking at the Hystrix Wiki, looks like that's what most
Hystrix integrations do (incl. the ones used to collect and store stats
into Graphite for example).
c) Decide if you want to store aggregations of stats from multiple app
instances (in that case understand how you can configure or 'fix' Turbine
to not alter the semantics of the original instance level stats, or
understand how/when to store the aggregated Turbine stats), or if it's
actually better to store stats from individual app instances... I'd
probably favor the latter, collect and store the individual instance data
in a DB and aggregate/interpret at rendering time later.
d) Investigate the CF firehose to see if it could help flow the metrics
you've collected to consumers that'll store them in your DBs; that firehose
will definitely be in the loop if you decide to flow the metrics with your
logs, then you can probably just connect a firehose nozzle to it that will
store the selected metrics to your DB.
On Thu, Nov 12, 2015 at 2:36 PM, KRuelY <kevinyudhiswara(a)gmail.com> wrote: