How about keying by criteria as well to know when to trigger a Webhook call? and maybe allow multiple registrations per URL? (e.g please call me back at http://foo.com/bar on new usage for org abc, and also when usage for app xyz goes over 1000, would probably be two separate registration docs for a single URL)

I think that would work well too. I guess the main thing between the two approaches is number of docs versus size of a doc. Building a query for either wouldn't be too hard, but putting multiple registrations inside a doc would probably add additional complexity to the notification logic.

+1 for a sort of quarantine on an unreachable Webhook. How about slow Webhooks causing back pressure problems? Would we quarantine these too?

This is kind of hard for me to figure out. If they continue to cause problems, leaving the slow webhooks would probably compound the problems as time goes on, but at the same time, they aren't quite in the same category as an unreachable one since they're actually reachable. Somehow the client needs to know their webhook is causing issues or unreachable.

- should we let the rating service app do this or have a separate notification service app?

I don't have a concrete opinion on this, but both sides have their merits. With a separate notification app, this keeps notification logic outside of something like aggregator such that aggregator stays doing aggregator logic and doesn't expand to something that it might not want to do. One thing to consider is dealing with the load of everything coming to a single notification app. I'm also not sure if this approach would design the logic in such a way that it won't be pluggable in any other app such that someone would not have to the notifications application running to use it.
I don't have as much to say on putting the logic inside of rating. The only thing I can bring up is my previous statement of having rating doing notification logic instead of strictly rating logic, and it does give a bigger separation of logic in terms of determining if a criteria is met or not.

- with partitioning of the orgs across multiple deployments of our apps (for scalability or regional deployments for example) do I need to first find the right service to register with (e.g. register with the us-south notification service or the eu-west notification service)? or can I register with a central notification service that will then figure out which deployment instance will call me back?

It might work better on a central notification service to prevent things like a client registering to an incorrect region. If we keep it separate and configured criteria continue to evolve, would a central notification service help along the lines of something like "notify if all my organizations across all the regions exceed X amount?"

- how do we secure the registration calls and Webhook callbacks?

The security on registration would probably just validate if the user's token has access to that organization/space/etc. For the webhook, I'm not sure if this falls within abacus.usage.read or abacus.usage.write, or if we would need a new scope to handle this case.

- do we replay notifications when we can't deliver them?

Ideally, it would make sense. Whether it's a simple series of retries or have replay logic that would later play through the set of unsent notifications. I think this is related to the quarantine logic. Some notifications when replayed will probably go through, some will fail and may continue to fail. Those that continue to fail may be due to the webhook their attached to, and thus, it may make sense to quarantine that webhook based upon these replay failures.

- can I register to receive all notifications from a certain point in the logical stream of notifications matching a criteria (e.g. call me back if this org consumed too much per hour at any point since last week)

I'm not sure if I completely understand this question. Is this pretty much setting a adhoc time range upon which notifications should be sent?