Re: Notifications on ORG, SPACE and USER modifications


Pablo Alonso Rodriguez <palonsoro@...>
 

For us, the main use case would also be security auditing, like for Mike.
We do not know yet what tool should receive them, but directly sending them
to a syslog endpoint (without having to pull them) may be a good
alternative (at least, among a bigger set of options).

Secondarily, it might also be interesting for billing.

2015-07-30 0:37 GMT+02:00 Piotr Przybylski <piotrp(a)us.ibm.com>:

We have several use cases
- in time event processing for usage, with notification instead of polling
we'll be able to keep up better, getting closer to real time for metering,
and thus rating and billing
- timely display of the new system objects like spaces or organizations,
being notified about new objects, avoids retrieving all objects and
comparing which is already known
- detecting and signaling entity field or property changes, for example
name changes, which can require polling and comparison
These in turn would allow better support uses like
- console or UI reactive to entity changes
- activity dashboards
- analytics

Piotr


[image: Inactive hide details for Mike Youngstrom ---07/29/2015 08:54:59
AM---For us the main use case is security auditing to keep a l]Mike
Youngstrom ---07/29/2015 08:54:59 AM---For us the main use case is security
auditing to keep a long term record of who has done anything.



From:


Mike Youngstrom <youngm(a)gmail.com>

To:


"Discussions about Cloud Foundry projects and the system overall." <
cf-dev(a)lists.cloudfoundry.org>

Date:


07/29/2015 08:54 AM

Subject:


Re: [cf-dev] Notifications on ORG, SPACE and USER modifications

Sent by:


cf-dev-bounces(a)lists.cloudfoundry.org
------------------------------



For us the main use case is security auditing to keep a long term record
of who has done anything. In the case of our Security team rather than use
CF events directly they preferred to have events forwarded to Security
Analytics. Today we pull events then forward the details to Security
Analytics via a syslog endpoint.

Mike

[0]
*http://www.emc.com/security/security-analytics/security-analytics.htm*
<http://www.emc.com/security/security-analytics/security-analytics.htm>

On Tue, Jul 28, 2015 at 11:26 PM, Dieu Cao <*dcao(a)pivotal.io*
<dcao(a)pivotal.io>> wrote:

Hi all interested in notifications on modification of resources,

It would be helpful for me in framing the "what" and the "why" of this
feature if you could also describe your specific use cases and pain points
on why you would want notifications on modifications and also which
resources you particularly care about.
Is it for real time updates on a dashboard? For consumption for
billing purposes? For triggering provisioning/deprovisioning of resources?

-Dieu

On Tue, Jul 28, 2015 at 11:05 AM, Jean-Sebastien Delfino <
*jsdelfino(a)gmail.com* <jsdelfino(a)gmail.com>> wrote:
I’m going to need something like this too for the CF Abacus service
metering project, as I’d like to track the lifecycle of orgs, users, etc to
match their history with the usage data reported for them.

Here’s a straw man description of what I had in mind:

- For Abacus, I’d need a Lossless API. Usage metering eventually
translates to billing and money, you don’t want to lose that :)

- An extension or variant of the current CF /v2/events API
supporting users, orgs, app usage etc, as even with a notification API I’ll
still need to do GETs sometimes.

- 304 responses with etags on these GETs (as suggested earlier in
the thread [1]) would be good.

- A Webhook style notification API where I could register interest
in a selection of events with a callback URL, and get these events POSTed
back to me at that URL, similar to what Github and many others do with
Webhooks.

- On top of Webhooks, it’d be nice to have a form of streaming
(either down to the client like the Firehose does or in the other direction
up to the Webhook callback URL), but I'm not sure if we’ll need that in the
project right away.

- We’d obviously need some form of security, maybe use my user
token to register for events on entities that I have access to?

- I’m also curious about the group’s thoughts on queueing and
back-pressure when events get generated faster that they can be consumed
for example. There was a mention of some message queuing earlier [2]. That
would make sense to me (although IMO it’d be good if the underlying MQ
didn’t shine through the API). What did you have in mind for this?

I guess there are quite a few things to figure out here! I’ll be
happy to collaborate with the community on these discussions.

Thoughts?

[1]
*http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p842.html*
<http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p842.html>
[2]
*http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p834.html*
<http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p834.html>

- Jean-Sebastien


On Fri, Jul 24, 2015 at 9:59 PM, Matt Cowger <*matt(a)cowger.us*
<matt(a)cowger.us>> wrote:
I think ETags is reasonable thought as well.

On Thu, Jul 23, 2015 at 4:39 PM, Benjamin Black <
*bblack(a)pivotal.io* <bblack(a)pivotal.io>> wrote:
ETags and a 304 response are specifically intended for that
purpose. I'd recommend that over relying on Last-Modified.


b

On Thu, Jul 23, 2015 at 12:34 AM, Koper, Dies <
*diesk(a)fast.au.fujitsu.com* <diesk(a)fast.au.fujitsu.com>>
wrote:
Or setting the Last-Modified HTTP response header
accordingly, and allow clients to use HTTP caching mechanisms
(Last-Modified, etc.) to get quick empty responses with the current APIs if
no changes have been made? (Or maybe this is already working so – haven’t
checked).



Regards,

Dies Koper



*From:* *cf-dev-bounces(a)lists.cloudfoundry.org*
<cf-dev-bounces(a)lists.cloudfoundry.org> [mailto:
*cf-dev-bounces(a)lists.cloudfoundry.org*
<cf-dev-bounces(a)lists.cloudfoundry.org>] *On Behalf Of *Matt
Cowger
* Sent:* Thursday, July 23, 2015 4:45 PM
* To:* Discussions about Cloud Foundry projects and the system overall.

*Subject:* Re: [cf-dev] Notifications on ORG, SPACE and
USER modifications



I've wanted something similar as well.



On a related note, having a CC API 'serial' number (for
each object in CC - apps, spaces, etc) that increments on every change
relevant to that object would be of value for detecting if something has
changed.



On Thu, Jul 23, 2015 at 3:27 PM, Dieu Cao <
*dcao(a)pivotal.io* <dcao(a)pivotal.io>> wrote:

There are a few different approaches to this and different
concerns that are possible.

The requests I've seen have been around wanting to be able
to subscribe to and filter the various events that cc currently generates
so that other behavior could be triggered.

We currently have events, app usage events, and service
usage events.

Is it acceptable for the notifications to be lossy?
Depends on the use case but If so, then the firehose may be an acceptable
approach.



The CAPI team is currently focusing on other work in the
near term, such as the v3 API and private brokers, but would be happy to
collaborate on a proposal.





On Wed, Jul 22, 2015 at 2:05 PM, Juan Pablo Genovese <
*juanpgenovese(a)gmail.com* <juanpgenovese(a)gmail.com>> wrote:

My take:



CC should have callbacks on for each model create, update
and delete methods. Those callbacks will send a message to an MQ, which you
can subscribe to consume those messages.

This can be expanded to pretty much every event we need to
track.

What do you think?



JP



2015-07-22 17:30 GMT-03:00 Matthias X Hub <
*matthias.hub(a)de.ibm.com* <matthias.hub(a)de.ibm.com>>:

Hi,

we (=IBM) are also having the need and are currently
investigating how to solve this. We plan to work on a proposal to discuss
this further with the cf community. I'll keep you updated on that.


Regards,
Matthias



From: Mike Youngstrom <*youngm(a)gmail.com*
<youngm(a)gmail.com>>
To: "Discussions about Cloud Foundry projects and
the system overall." <*cf-dev(a)lists.cloudfoundry.org*
<cf-dev(a)lists.cloudfoundry.org>>
Date: 22.07.2015 20:57
Subject: Re: [cf-dev] Notifications on ORG, SPACE
and USER modifications
Sent by: *cf-dev-bounces(a)lists.cloudfoundry.org*
<cf-dev-bounces(a)lists.cloudfoundry.org>

------------------------------




We have the same need. Today we are polling the CC.

It would be nice for us also if we could get CC event
notifications via something like the firehose.

Mike

On Wed, Jul 22, 2015 at 10:23 AM, Juan Pablo Genovese <
*juanpgenovese(a)gmail.com* <juanpgenovese(a)gmail.com>>
wrote:
I mean, I know you can list those events thru the API, but
I want something that will react on an event instead of having to be
constantly polling for them.

2015-07-22 13:18 GMT-03:00 Juan Pablo Genovese <
*juanpgenovese(a)gmail.com* <juanpgenovese(a)gmail.com>>:
Sree,

thanks! Any pointers on how can I hook up to these audit
events?

Thank you!

2015-07-22 13:12 GMT-03:00 Sree Tummidi <
*stummidi(a)pivotal.io* <stummidi(a)pivotal.io>>:
I believe there are audit events generated for all these
actions which can be captured and forwarded to an SIEM solution like splunk

Thanks,
Sree

Sent from my iPhone

On Jul 22, 2015, at 8:54 AM, Juan Pablo Genovese <
*juanpgenovese(a)gmail.com* <juanpgenovese(a)gmail.com>> wrote:

Guys,

I need to somehow hook up into the Cloud Controller (CC)
to capture ORG, SPACE and USER deletion, insertion and update.

So far, I considered some approaches, such as forking the
CC (the least favorite) and modifying the code with some hooks, tapping
into Nginx to capture the requests, and using triggers in the database to
capture each event and send the necessary info to a service.

What do you think?
Any other idea you might have?

Thanks!

--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------

*http://www.jpgenovese.com* <http://www.jpgenovese.com/>
_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>

_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>




--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
*http://www.jpgenovese.com* <http://www.jpgenovese.com/>



--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
*http://www.jpgenovese.com* <http://www.jpgenovese.com/>


_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>

_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>


_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>




--

Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------

*http://www.jpgenovese.com* <http://www.jpgenovese.com/>


_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>




_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>




--

-- Matt

_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>



_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>



--
-- Matt

_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>



--
Jean-Sebastien

Sent from my DynaTAC 8000x

_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>



_______________________________________________
cf-dev mailing list
*cf-dev(a)lists.cloudfoundry.org* <cf-dev(a)lists.cloudfoundry.org>
*https://lists.cloudfoundry.org/mailman/listinfo/cf-dev*
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.