Hi,

We're trying to fix Abacus issue 88: Missing aggregated usage for the running application [1].

Background
=========

See the jsdelfino comment in the GitHub issue [2]. TL;DR: Resource providers have to send a 'ping' doc per month for time-based metrics.

Proposed solution
==============

We decided to implement a solution in Abacus that frees the usage providers from sending the 'ping' submission.

To fix the issue we decided to:
1. Distinguish between time-based (linux-container) and discrete usage metrics (the rest basically)
2. Store the time-based metrics in a separate DB(s)

We already drafted a proposal for adding measurement type in the usage plans with PR #320 [3].

We're about to spike on storing the time-based metrics in their own Database, but we wanted to get the community opinion on the topic.

Motivation
========

The discrete usage submitted to Abacus is:
* stored in partitioned databases, due to their size/number
* like an event log, storing the history of the usage/resources

In contrast the current time-based metrics are:
* limited number (usually around 2 million on a loaded CF system)
* storing just the app resources usage state (GB/h consumed so far, GB/h consuming currently)

Therefore it looks like a good idea to separate the two usage metrics types and store the time-based metrics in a separate database. This will allow us not only to solve the issue, but also to store and query the data more effectively.

We may still need to maintain 2 databases and swap new/old (irrelevant) metrics to reduce the DB size on the month boundaries.


Regards,
Hristo & Adriana

[1] https://github.com/cloudfoundry-incubator/cf-abacus/issues/88
[2] https://github.com/cloudfoundry-incubator/cf-abacus/issues/88#issuecomment-148498164
[3] https://github.com/cloudfoundry-incubator/cf-abacus/pull/320