Hi,

One of the thing that I want to do is to persist the metrics performance
collected by abacus-perf. What would be the best way to do this? I've been
through some solutions, but none of them seems to be the "correct" solution.

The scenario is this: I have an application running, and there are 2
instances of this application currently running.

To collect the metrics performance of my application, I need to aggregate
the metrics data collected by each instance's abacus-perf and store them in
a database.

The first solution is to use Turbine. Using Eureka to keep track each
instance's ip address, I can configure Turbine to use Eureka instance
discovery. This way turbine will have aggregated metrics data collected by
each instance's abacus-perf. The next thing to do is to have a separate
application 'peeks' at the turbine stream at some interval and post them to
database. The problem with this is that Turbine persists the metrics data
when there are no activity in the application, and it will flush the metrics
data when a new stats come in. Meaning that every time I peek into the
turbine stream, I have to check if I already posted these data to the
database.

The second solution is to have each instance post independently. By using
abacus-perf's 'all()' I can set an interval that would call all(), check the
timewindow, and post accordingly. The restriction is that I can only post
the previous timewindow (since the current window is not yet done), and I
need to filter 0 data. Another restriction is that my interval cannot exceed
perf's interval. The problem with this is that
I am playing with the time interval. There would be some occasion that I
might lose some data. I'm not sure that this would cover the time where perf
flushes out old metrics when a new one comes in. I need to make sure that I
save the data before perf flushes.

Another solution is to mimic what the hystrix module is doing: Instead of
streaming the metrics to the hystrix dashboard, I would post to the
database. I have yet to try this solution.

Currently I'm not sure what is the best way to persist the metrics
performance collected by the abacus-perf with accuracy, and I would like to
have some inputs/suggestion on how to persist the metrics. Thanks!





--
View this message in context: http://cf-dev.70369.x6.nabble.com/abacus-perf-Persisting-Metrics-performance-tp2693.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi Kevin,

One of the thing that I want to do is to persist the metrics performance

collected by abacus-perf.

Interesting feature! It'd be good to understand what you're trying to do
with that data (I think Assk for asking a similar question) as that'll help
us provide better implementation suggestions.

The first solution is to use Turbine...

...

The problem with this is that Turbine persists the metrics data

when there are no activity in the application, and it will flush the metrics
data when a new stats come in.

Not sure I'm following here. Can you give a bit more details to help us
understand how Turbine alters the stats and what problems that causes in
your collection + store logic?

Another solution is to mimic what the hystrix module is doing: Instead of streaming

the metrics to the hystrix dashboard, I would post to the database.

The Abacus-hystrix module responds to GET /hystrix.stream requests, and
doesn't do anything unless a monitor requests the stats. I'm not sure that
pro-actively posting the stats to a DB from each app instance will work so
well... as IMO that'll generate a lot of DB traffic from all these app
instances, will slow down these apps, and won't give you an aggregated
stats at the app level anyway (more on that below, however).

Here's a few more suggestions for you:

a) Give us a bit more context on how you intend to use the data you're
storing... if this is for use with Graphite for example, there's already a
number of blog posts out there that cover that; if you'd like to store the
data in ELK for searching then you might want to log these metrics and flow
them to ELK as part of your logs; if you'd like to store the data in your
own DB and render it using custom made dashboards later then we can explore
other solutions...

b) Try to leverage the current flow (with app instances providing stats on
demand at a /hystrix.stream endpoint and an external monitoring app
collecting these stats) rather than creating yet another completely
different flow; looking at the Hystrix Wiki, looks like that's what most
Hystrix integrations do (incl. the ones used to collect and store stats
into Graphite for example).

c) Decide if you want to store aggregations of stats from multiple app
instances (in that case understand how you can configure or 'fix' Turbine
to not alter the semantics of the original instance level stats, or
understand how/when to store the aggregated Turbine stats), or if it's
actually better to store stats from individual app instances... I'd
probably favor the latter, collect and store the individual instance data
in a DB and aggregate/interpret at rendering time later.

d) Investigate the CF firehose to see if it could help flow the metrics
you've collected to consumers that'll store them in your DBs; that firehose
will definitely be in the loop if you decide to flow the metrics with your
logs, then you can probably just connect a firehose nozzle to it that will
store the selected metrics to your DB.

HTH

- Jean-Sebastien

On Thu, Nov 12, 2015 at 2:36 PM, KRuelY <kevinyudhiswara(a)gmail.com> wrote:

Hi,

One of the thing that I want to do is to persist the metrics performance
collected by abacus-perf. What would be the best way to do this? I've been
through some solutions, but none of them seems to be the "correct"
solution.

The scenario is this: I have an application running, and there are 2
instances of this application currently running.

To collect the metrics performance of my application, I need to aggregate
the metrics data collected by each instance's abacus-perf and store them in
a database.

The first solution is to use Turbine. Using Eureka to keep track each
instance's ip address, I can configure Turbine to use Eureka instance
discovery. This way turbine will have aggregated metrics data collected by
each instance's abacus-perf. The next thing to do is to have a separate
application 'peeks' at the turbine stream at some interval and post them to
database. The problem with this is that Turbine persists the metrics data
when there are no activity in the application, and it will flush the
metrics
data when a new stats come in. Meaning that every time I peek into the
turbine stream, I have to check if I already posted these data to the
database.

The second solution is to have each instance post independently. By using
abacus-perf's 'all()' I can set an interval that would call all(), check
the
timewindow, and post accordingly. The restriction is that I can only post
the previous timewindow (since the current window is not yet done), and I
need to filter 0 data. Another restriction is that my interval cannot
exceed
perf's interval. The problem with this is that
I am playing with the time interval. There would be some occasion that I
might lose some data. I'm not sure that this would cover the time where
perf
flushes out old metrics when a new one comes in. I need to make sure that I
save the data before perf flushes.

Another solution is to mimic what the hystrix module is doing: Instead of
streaming the metrics to the hystrix dashboard, I would post to the
database. I have yet to try this solution.

Currently I'm not sure what is the best way to persist the metrics
performance collected by the abacus-perf with accuracy, and I would like to
have some inputs/suggestion on how to persist the metrics. Thanks!





--
View this message in context:
http://cf-dev.70369.x6.nabble.com/abacus-perf-Persisting-Metrics-performance-tp2693.html
Sent from the CF Dev mailing list archive at Nabble.com.

In my opinion, writing to a database at the source of the collected data

would drastically reduce the programming complexity and would help to make
the data more consistent with the source.

That approach would also create another path for these stats (apps
proactively pushing stats to DB, then monitor pulling from DB) on top of
the current one (monitor pulling from apps returning the stats). So, I'd
argue that having two significantly different paths for these stats instead
of one is actually adding complexity rather than reducing complexity :).

However, I always wonder why one would need to persist this data. any

reasons?

I have the same question. Would like to understand Kevin's use case a bit
better.


- Jean-Sebastien

On Thu, Nov 12, 2015 at 3:26 PM, Saravanakumar A Srinivasan <
sasrin(a)us.ibm.com> wrote:

I would like to add one more to the list of possible solutions for further
discussion:

How about extending abacus-perf to optionally persist collected
performance metrics into a database?
In my opinion, writing to a database at the source of the collected data
would drastically reduce the programming complexity and would help to make
the data more consistent with the source.

However, I always wonder why one would need to persist this data. any
reasons?


Thanks,
Saravanakumar Srinivasan (Assk),


-----KRuelY <kevinyudhiswara(a)gmail.com> wrote: -----
To: cf-dev(a)lists.cloudfoundry.org
From: KRuelY <kevinyudhiswara(a)gmail.com>
Date: 11/12/2015 02:45PM
Subject: [cf-dev] [abacus-perf] Persisting Metrics performance


Hi,

One of the thing that I want to do is to persist the metrics performance
collected by abacus-perf. What would be the best way to do this? I've been
through some solutions, but none of them seems to be the "correct"
solution.

The scenario is this: I have an application running, and there are 2
instances of this application currently running.

To collect the metrics performance of my application, I need to aggregate
the metrics data collected by each instance's abacus-perf and store them in
a database.

The first solution is to use Turbine. Using Eureka to keep track each
instance's ip address, I can configure Turbine to use Eureka instance
discovery. This way turbine will have aggregated metrics data collected by
each instance's abacus-perf. The next thing to do is to have a separate
application 'peeks' at the turbine stream at some interval and post them to
database. The problem with this is that Turbine persists the metrics data
when there are no activity in the application, and it will flush the
metrics
data when a new stats come in. Meaning that every time I peek into the
turbine stream, I have to check if I already posted these data to the
database.

The second solution is to have each instance post independently. By using
abacus-perf's 'all()' I can set an interval that would call all(), check
the
timewindow, and post accordingly. The restriction is that I can only post
the previous timewindow (since the current window is not yet done), and I
need to filter 0 data. Another restriction is that my interval cannot
exceed
perf's interval. The problem with this is that
I am playing with the time interval. There would be some occasion that I
might lose some data. I'm not sure that this would cover the time where
perf
flushes out old metrics when a new one comes in. I need to make sure that I
save the data before perf flushes.

Another solution is to mimic what the hystrix module is doing: Instead of
streaming the metrics to the hystrix dashboard, I would post to the
database. I have yet to try this solution.

Currently I'm not sure what is the best way to persist the metrics
performance collected by the abacus-perf with accuracy, and I would like to
have some inputs/suggestion on how to persist the metrics. Thanks!





--
View this message in context:
http://cf-dev.70369.x6.nabble.com/abacus-perf-Persisting-Metrics-performance-tp2693.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi all,

Thanks for the suggestions. I will look at the suggestions, and think about
them more. The reason of saving to database and persisting them is to see
how my apps are doing at the specific time. Specifically when I'm asleep and
my apps are having problems, I would like to know which requests are
failing, why they're failing, and have concrete data to blame my
dependencies . Joke aside, my goal is what I have said. I would like to
persist the data.


I believe my tasks right now are:
1. to make my apps use breaker (with breaking turned off)
2. save to database.

One issue I know is that the name doesn't shows up correctly. It shows as
wrapper. Right now I'll not think about it, and focus on making my apps use
perf. Thanks!



--
View this message in context: http://cf-dev.70369.x6.nabble.com/abacus-perf-Persisting-Metrics-performance-tp2693p2728.html
Sent from the CF Dev mailing list archive at Nabble.com.