We have several use cases
- in time event processing for usage, with notification instead of polling
we'll be able to keep up better, getting closer to real time for metering,
and thus rating and billing
- timely display of the new system objects like spaces or organizations,
being notified about new objects, avoids retrieving all objects and
comparing which is already known
- detecting and signaling entity field or property changes, for example
name changes, which can require polling and comparison
These in turn would allow better support uses like
- console or UI reactive to entity changes
- activity dashboards
- analytics

Piotr



|------------>
| From: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Mike Youngstrom <youngm(a)gmail.com> |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|"Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org> |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|07/29/2015 08:54 AM |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Re: [cf-dev] Notifications on ORG, SPACE and USER modifications |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Sent by: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|cf-dev-bounces(a)lists.cloudfoundry.org |
>--------------------------------------------------------------------------------------------------------------------------------------------------|





For us the main use case is security auditing to keep a long term record of
who has done anything.  In the case of our Security team rather than use CF
events directly they preferred to have events forwarded to Security
Analytics.  Today we pull events then forward the details to Security
Analytics via a syslog endpoint.

Mike

[0] http://www.emc.com/security/security-analytics/security-analytics.htm

On Tue, Jul 28, 2015 at 11:26 PM, Dieu Cao <dcao(a)pivotal.io> wrote:
Hi all interested in notifications on modification of resources,

It would be helpful for me in framing the "what" and the "why" of this
feature if you could also describe your specific use cases and pain
points on why you would want notifications on modifications and also
which resources you particularly care about.
Is it for real time updates on a dashboard? For consumption for billing
purposes?  For triggering provisioning/deprovisioning of resources?

-Dieu

On Tue, Jul 28, 2015 at 11:05 AM, Jean-Sebastien Delfino <
jsdelfino(a)gmail.com> wrote:
I’m going to need something like this too for the CF Abacus service
metering project, as I’d like to track the lifecycle of orgs, users, etc
to match their history with the usage data reported for them.

Here’s a straw man description of what I had in mind:

- For Abacus, I’d need a Lossless API. Usage metering eventually
translates to billing and money, you don’t want to lose that :)

- An extension or variant of the current CF /v2/events API supporting
users, orgs, app usage etc, as even with a notification API I’ll still
need to do GETs sometimes.

- 304 responses with etags on these GETs (as suggested earlier in the
thread [1]) would be good.

- A Webhook style notification API where I could register interest in a
selection of events with a callback URL, and get these events POSTed
back to me at that URL, similar to what Github and many others do with
Webhooks.

- On top of Webhooks, it’d be nice to have a form of streaming (either
down to the client like the Firehose does or in the other direction up
to the Webhook callback URL), but I'm not sure if we’ll need that in the
project right away.

- We’d obviously need some form of security, maybe use my user token to
register for events on entities that I have access to?

- I’m also curious about the group’s thoughts on queueing and
back-pressure when events get generated faster that they can be consumed
for example. There was a mention of some message queuing earlier [2].
That would make sense to me (although IMO it’d be good if the underlying
MQ didn’t shine through the API). What did you have in mind for this?

I guess there are quite a few things to figure out here! I’ll be happy
to collaborate with the community on these discussions.

Thoughts?

[1]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p842.html
[2]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p834.html

- Jean-Sebastien



On Fri, Jul 24, 2015 at 9:59 PM, Matt Cowger <matt(a)cowger.us> wrote:
I think ETags is reasonable thought as well.

On Thu, Jul 23, 2015 at 4:39 PM, Benjamin Black <bblack(a)pivotal.io>
wrote:
ETags and a 304 response are specifically intended for that purpose.
I'd recommend that over relying on Last-Modified.


b

On Thu, Jul 23, 2015 at 12:34 AM, Koper, Dies <
diesk(a)fast.au.fujitsu.com> wrote:
Or setting the Last-Modified HTTP response header accordingly, and
allow clients to use HTTP caching mechanisms (Last-Modified, etc.)
to get quick empty responses with the current APIs if no changes
have been made? (Or maybe this is already working so – haven’t
checked).





Regards,


Dies Koper





From: cf-dev-bounces(a)lists.cloudfoundry.org [mailto:
cf-dev-bounces(a)lists.cloudfoundry.org] On Behalf Of Matt Cowger
Sent: Thursday, July 23, 2015 4:45 PM
To: Discussions about Cloud Foundry projects and the system
overall.


Subject: Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications





I've wanted something similar as well.





On a related note, having a CC API 'serial' number (for each object
in CC - apps, spaces, etc) that increments on every change relevant
to that object would be of value for detecting if something has
changed.





On Thu, Jul 23, 2015 at 3:27 PM, Dieu Cao <dcao(a)pivotal.io> wrote:


There are a few different approaches to this and different concerns
that are possible.


The requests I've seen have been around wanting to be able to
subscribe to and filter the various events that cc currently
generates so that other behavior could be triggered.


We currently have events, app usage events, and service usage
events.


Is it acceptable for the notifications to be lossy?  Depends on the
use case but If so, then the firehose may be an acceptable
approach.





The CAPI team is currently focusing on other work in the near term,
such as the v3 API and private brokers, but would be happy to
collaborate on a proposal.








On Wed, Jul 22, 2015 at 2:05 PM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:


My take:





CC should have callbacks on for each model create, update and
delete methods. Those callbacks will send a message to an MQ, which
you can subscribe to consume those messages.


This can be expanded to pretty much every event we need to track.


What do you think?





JP





2015-07-22 17:30 GMT-03:00 Matthias X Hub <matthias.hub(a)de.ibm.com
>:


Hi,

we (=IBM) are also having the need and are currently investigating
how to solve this. We plan to work on a proposal to discuss this
further with the cf community. I'll keep you updated on that.

Regards,
Matthias



From:        Mike Youngstrom <youngm(a)gmail.com>
To:        "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date:        22.07.2015 20:57
Subject:        Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications
Sent by:        cf-dev-bounces(a)lists.cloudfoundry.org








We have the same need.  Today we are polling the CC.

It would be nice for us also if we could get CC event notifications
via something like the firehose.

Mike

On Wed, Jul 22, 2015 at 10:23 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:
I mean, I know you can list those events thru the API, but I want
something that will react on an event instead of having to be
constantly polling for them.

2015-07-22 13:18 GMT-03:00 Juan Pablo Genovese <
juanpgenovese(a)gmail.com>:
Sree,

thanks! Any pointers on how can I hook up to these audit events?

Thank you!

2015-07-22 13:12 GMT-03:00 Sree Tummidi <stummidi(a)pivotal.io>:
I believe there are audit events generated for all these actions
which can be captured and forwarded to an SIEM solution like splunk


Thanks,
Sree

Sent from my iPhone

On Jul 22, 2015, at 8:54 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:

Guys,

I need to somehow hook up into the Cloud Controller (CC) to capture
ORG, SPACE and USER deletion, insertion and update.

So far, I considered some approaches, such as forking the CC (the
least favorite) and modifying the code with some hooks, tapping
into Nginx to capture the requests, and using triggers in the
database to capture each event and send the necessary info to a
service.

What do you think?
Any other idea you might have?

Thanks!

--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------


http://www.jpgenovese.com
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com



--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev









--


Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------


http://www.jpgenovese.com



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev






_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev









--


-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Jean-Sebastien

Sent from my DynaTAC 8000x

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev