Re: Billing & Metering of app usage with Abacus


CF Runtime
 

Piotr,

The timestamps not being correct is a known limitation of how events are
being generated, but as you said, order should be guaranteed (and
timestamps should hopefully be close).

Duplicate events are something I'm not aware of though. In theory only one
API instance should be able to get a database lock on an app, and should
not release it until it is done updating and has recorded the app usage
event. Do you have any details on what caused duplicate events?

Joseph
OSS Release Integration Team

On Wed, Aug 12, 2015 at 8:07 AM, Piotr Przybylski <piotrp(a)us.ibm.com> wrote:

Hi, I am also looking at runtime submissions for Abacus, worked on it for
Bluemix, couple of points for discussion.

In addition to usage events (start followed by stop), the scaling and
duplicate events need to be handled. The former is a START followed by
START with memory or instance count changed, the latter can be STOP
followed by STOP.
We also encountered a situation where the ordering is correct - START
followed by STOP but the timestamp for the START is later than STOP.

For the running applications - your points #2 and #3 are a working though
it may generate fair amount of traffic, depending on frequency and number
of running applications. Eventually we may want to look at alternatives,
for example enhance metering to allow for time based submissions. Instead
of continually submitting usage, submit the state of application -
(STARTED, memory, instances), the metering could then calculate usage for
that application based on the time passed until application is stopped.

I think handling some of above scenarios requires persistence, even if to
log CF events that were used for submission (or skipped). You may want to
persist state to recovery from application failure or restart, as well as
keep track of running/active applications.

Thanks,

Piotr


Hristo Iliev <hsiliev(a)gmail.com> wrote on 08/07/2015 08:07:53 AM:

We want to integrate the metrics provided by Cloud Foundry with Abacus.
We plan to create a billing/metering integration layer that:
fetches the app usage events from CC
inserts runtime usage data to Abacus
The events should be processed to build usage data based on them.
This is much like the idea outlined in Dr.Nic's blog. The
integration layer can do this periodically.
AFAIK Abacus provides usage reports for the current month only and
not for arbitrary period of time. This implies some restrictions to
what we can report when the application is:
1. started and stopped several times in the month
2. started in the current month but not stopped at all
3. started in a previous month and not stopped
The first problem can be solved by iterating through the events and
finding the respective start and end timestamps that have to be
reported to Abacus.


The second issue might be solved by reporting small amounts of
usage, stating from the last start event and continuing to report on
every poll of the integration layer. For example we can report several
usages:
start: 1438945112; end: 1438946000 (current time for the billing
integration)
start: 1438946000 (previous reporting cycle); end: 1438947000
The third issue might be solved by finding the last start event and
reporting in the same manner as with #2.
Reporting usage in small steps might require persistence so we can
store the end time of the previous reporting. We might use in-memory
cache and reach to Abacus as primary storage. If Abacus can
accumulate usage reporting we can even get rid of the persistence and
cache.


Is such integration in the scope of Abacus project?

Regards,
Hristo Iliev

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.