[abacus] Separate time-based from discrete usage metrics
We're trying to fix Abacus issue 88: Missing aggregated usage for the running application .
See the jsdelfino comment in the GitHub issue . TL;DR: Resource providers have to send a 'ping' doc per month for time-based metrics.
We decided to implement a solution in Abacus that frees the usage providers from sending the 'ping' submission.
To fix the issue we decided to:
1. Distinguish between time-based (linux-container) and discrete usage metrics (the rest basically)
2. Store the time-based metrics in a separate DB(s)
We already drafted a proposal for adding measurement type in the usage plans with PR #320 .
We're about to spike on storing the time-based metrics in their own Database, but we wanted to get the community opinion on the topic.
The discrete usage submitted to Abacus is:
* stored in partitioned databases, due to their size/number
* like an event log, storing the history of the usage/resources
In contrast the current time-based metrics are:
* limited number (usually around 2 million on a loaded CF system)
* storing just the app resources usage state (GB/h consumed so far, GB/h consuming currently)
Therefore it looks like a good idea to separate the two usage metrics types and store the time-based metrics in a separate database. This will allow us not only to solve the issue, but also to store and query the data more effectively.
We may still need to maintain 2 databases and swap new/old (irrelevant) metrics to reduce the DB size on the month boundaries.
Hristo & Adriana