Re: Notifications on ORG, SPACE and USER modifications


Jean-Sebastien Delfino
 

I’m going to need something like this too for the CF Abacus service
metering project, as I’d like to track the lifecycle of orgs, users, etc to
match their history with the usage data reported for them.


Here’s a straw man description of what I had in mind:


- For Abacus, I’d need a Lossless API. Usage metering eventually translates
to billing and money, you don’t want to lose that :)


- An extension or variant of the current CF /v2/events API supporting
users, orgs, app usage etc, as even with a notification API I’ll still need
to do GETs sometimes.


- 304 responses with etags on these GETs (as suggested earlier in the
thread [1]) would be good.


- A Webhook style notification API where I could register interest in a
selection of events with a callback URL, and get these events POSTed back
to me at that URL, similar to what Github and many others do with Webhooks.


- On top of Webhooks, it’d be nice to have a form of streaming (either down
to the client like the Firehose does or in the other direction up to the
Webhook callback URL), but I'm not sure if we’ll need that in the project
right away.


- We’d obviously need some form of security, maybe use my user token to
register for events on entities that I have access to?


- I’m also curious about the group’s thoughts on queueing and back-pressure
when events get generated faster that they can be consumed for example.
There was a mention of some message queuing earlier [2]. That would make
sense to me (although IMO it’d be good if the underlying MQ didn’t shine
through the API). What did you have in mind for this?


I guess there are quite a few things to figure out here! I’ll be happy to
collaborate with the community on these discussions.


Thoughts?


[1]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p842.html

[2]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p834.html


- Jean-Sebastien

On Fri, Jul 24, 2015 at 9:59 PM, Matt Cowger <matt(a)cowger.us> wrote:

I think ETags is reasonable thought as well.

On Thu, Jul 23, 2015 at 4:39 PM, Benjamin Black <bblack(a)pivotal.io> wrote:

ETags and a 304 response are specifically intended for that purpose. I'd
recommend that over relying on Last-Modified.


b

On Thu, Jul 23, 2015 at 12:34 AM, Koper, Dies <diesk(a)fast.au.fujitsu.com>
wrote:

Or setting the Last-Modified HTTP response header accordingly, and
allow clients to use HTTP caching mechanisms (Last-Modified, etc.) to get
quick empty responses with the current APIs if no changes have been made?
(Or maybe this is already working so – haven’t checked).



Regards,

Dies Koper



*From:* cf-dev-bounces(a)lists.cloudfoundry.org [mailto:
cf-dev-bounces(a)lists.cloudfoundry.org] *On Behalf Of *Matt Cowger
*Sent:* Thursday, July 23, 2015 4:45 PM
*To:* Discussions about Cloud Foundry projects and the system overall.
*Subject:* Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications



I've wanted something similar as well.



On a related note, having a CC API 'serial' number (for each object in
CC - apps, spaces, etc) that increments on every change relevant to that
object would be of value for detecting if something has changed.



On Thu, Jul 23, 2015 at 3:27 PM, Dieu Cao <dcao(a)pivotal.io> wrote:

There are a few different approaches to this and different concerns that
are possible.

The requests I've seen have been around wanting to be able to subscribe
to and filter the various events that cc currently generates so that other
behavior could be triggered.

We currently have events, app usage events, and service usage events.

Is it acceptable for the notifications to be lossy? Depends on the use
case but If so, then the firehose may be an acceptable approach.



The CAPI team is currently focusing on other work in the near term, such
as the v3 API and private brokers, but would be happy to collaborate on a
proposal.





On Wed, Jul 22, 2015 at 2:05 PM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:

My take:



CC should have callbacks on for each model create, update and delete
methods. Those callbacks will send a message to an MQ, which you can
subscribe to consume those messages.

This can be expanded to pretty much every event we need to track.

What do you think?



JP



2015-07-22 17:30 GMT-03:00 Matthias X Hub <matthias.hub(a)de.ibm.com>:

Hi,

we (=IBM) are also having the need and are currently investigating how
to solve this. We plan to work on a proposal to discuss this further with
the cf community. I'll keep you updated on that.

Regards,
Matthias



From: Mike Youngstrom <youngm(a)gmail.com>
To: "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date: 22.07.2015 20:57
Subject: Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications
Sent by: cf-dev-bounces(a)lists.cloudfoundry.org
------------------------------




We have the same need. Today we are polling the CC.

It would be nice for us also if we could get CC event notifications via
something like the firehose.

Mike

On Wed, Jul 22, 2015 at 10:23 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:
I mean, I know you can list those events thru the API, but I want
something that will react on an event instead of having to be constantly
polling for them.

2015-07-22 13:18 GMT-03:00 Juan Pablo Genovese <juanpgenovese(a)gmail.com>:

Sree,

thanks! Any pointers on how can I hook up to these audit events?

Thank you!

2015-07-22 13:12 GMT-03:00 Sree Tummidi <stummidi(a)pivotal.io>:
I believe there are audit events generated for all these actions which
can be captured and forwarded to an SIEM solution like splunk

Thanks,
Sree

Sent from my iPhone

On Jul 22, 2015, at 8:54 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:

Guys,

I need to somehow hook up into the Cloud Controller (CC) to capture ORG,
SPACE and USER deletion, insertion and update.

So far, I considered some approaches, such as forking the CC (the least
favorite) and modifying the code with some hooks, tapping into Nginx to
capture the requests, and using triggers in the database to capture each
event and send the necessary info to a service.

What do you think?
Any other idea you might have?

Thanks!

--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------

http://www.jpgenovese.com
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com



--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev





--

Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------

http://www.jpgenovese.com


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev





--

-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Jean-Sebastien

Sent from my DynaTAC 8000x

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.