Date   

Re: FYI: CAB call for July 2015 - moved to 7/15 @ 8a PDT

Guillaume Berche
 

Thanks again to Phil Whelan for crafting the greatly detailed cab minutes.

http://www.activestate.com/blog/2015/07/cloud-foundry-advisory-board-meeting-2015-july


The MEGA section says the following:

*Updates from this MEGA team for the past week include having extracted
etcd-release and now they are working on extracting consul-release, setting
up pipelines for them, testing them and publishing them to bosh.io
<http://bosh.io/>.*

*Longer-term they want to generalize the patterns that have been introduced
for Diego around manifest generation.*

*Cornelia Davis from Pivotal asked Amit to confirm whether manifest
generation was moving away from Spiff, to which Amit replied "yes"."*

Looking at diego repo, I still see the spiff installation instructions, and
spiff used for manifest generation [2]. Where can I learn about the
manifest generation pattern for diego, to better understand this CAB
discussion ?

Thanks in advance,

Guillaume.

[2]
https://github.com/cloudfoundry-incubator/diego-release/blob/develop/scripts/generate-deployment-manifest#L84-L103

On Tue, Jul 14, 2015 at 4:56 PM, Michael Maximilien <maxim(a)us.ibm.com>
wrote:

FYI...

Final reminder of the CAB call tomorrow at 8a PDT. See all info from last
week's email below.

Also reminder to all PMs to include their highlights and to others to list
any proposals or discussion items.

Talk to you all tomorrow morning. Best,

dr.max
ibm cloud labs
silicon valley, ca

Sent from my iPhone

On Jul 7, 2015, at 9:54 AM, Michael Maximilien <maxim(a)us.ibm.com> wrote:

Hi, all,

After conferring with James and Chip, we've agreed to move the call to
next week Wednesday @ same time (8a PDT).

This helps folks who have extended their July 4th holiday in U.S. and
hopefully is not inconvenient to others.

Finally, reminders:

1. Note the call information is relatively new (same as last time). Please
take note:

USA 888-426-6840; 215-861-6239 *| *Participant code: *1985291*
*Other countries: *http://goo.gl/RnNfc1

2. Please add to the agenda of you have something to discuss. I will send
reminders again early next week.

Agenda:
https://docs.google.com/document/d/1SCOlAquyUmNM-AQnekCOXiwhLs6gveTxAcduvDcW_xI/edit

3. If you own any of the recurring meetings please update with above.

Thanks much. Best,

dr.max
ibm cloud labs
silicon valley, ca

dr.max
ibm cloud labs
silicon valley, ca

Sent from my iPhone


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Utilities PMC - 2015-07-07 notes

Mike Dalessio
 

Sorry all,

The correct URL for LicenseFinder tracker is:
https://www.pivotaltracker.com/n/projects/234851

However, the Toolsmiths tracker is private as there's quite a bit of work
in there that's specific to Pivotal, so I apologize for attempting to link
to it.

On Thu, Jul 9, 2015 at 1:05 AM, Guillaume Berche <bercheg(a)gmail.com> wrote:

Thanks Mike for the detailed notes.

It seems the toolsmiths-tracker url is wrong and pointing to the
buildpacks tracker, and the IF-tracker has not yet been configured for
being public (I get an "You are not allowed to view the requested page.")

[toolsmiths-tracker]: https://www.pivotaltracker.com/n/projects/1042066
[lf-tracker]: https://www.pivotaltracker.com/n/projects/939266

Thanks in advance,

Guillaume.

On Wed, Jul 8, 2015 at 4:58 PM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Hi all,

We had a meeting of the Utilities PMC yesterday, permanent notes are at:


https://github.com/cloudfoundry/pmc-notes/blob/master/Utilities/2015-07-07-utilities.md

which I've helpfully copied into this email below.

Cheers,
-mike


-----

*# Utilities PMC Meeting 2015-07-07*

*## Agenda*

1. Update on CI tools (Mike Dalessio)
2. Update on CLI (Greg Oehmen)
3. Update on Eclipse plugin and Java tools (Ryan Morgan)
4. Proposal to accept receptor-client into incubation
5. Open Discussion


*## Attendees*

* Chip Childers, Cloud Foundry Foundation
* Mike Dalessio, Pivotal (PMC lead)
* Deepak Vij, Huawei


*## Update on CI tools (Mike Dalessio)*

- Krafa track of work has been broadly defined, and it's in the
Toolsmiths [public backlog and icebox][toolsmiths-tracker].

- A pair is continuing work on [License Finder][lf], which is an OSS tool
to
examine a project's declared dependencies (via a package manager)
and discover licenses and changes in licensing. This should be
useful for the Foundation as well. (See [Tracker here][lf-tracker].)


*## Update on CLI (Greg Oehman)*

The CLI team released version 6.12.1 on 7/6/2015. Release notes,
download instructions and binaries are available at
https://github.com/cloudfoundry/cli/releases


*## Update on Eclipse plugin and Java tools (Ryan Morgan)*

* Eclipse tooling 1.8.3 release (Nieraj posted notes about this to
cf-dev).


*## Proposal to accept receptor-client into incubation*

Last meeting, Ryan Morgan [proposed receptor-client for
acceptance][receptor-client-proposal] into incubation.



No objections having been raised, on either the mailing list or in the
PMC meeting, this is now accepted into incubation.


*## Open Discussion*

None.


-----

Links:

[toolsmiths-tracker]: https://www.pivotaltracker.com/n/projects/1042066
[lf-tracker]: https://www.pivotaltracker.com/n/projects/939266
[lf]: https://github.com/pivotal/LicenseFinder
[receptor-client-proposal]:
http://cf-dev.70369.x6.nabble.com/cf-dev-Proposal-for-incubation-of-receptor-client-Java-client-for-receptor-td440.html


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Utilities PMC - 2015-07-07 notes

Guillaume Berche
 

Mike, not sure you had seen my question to the utilities pointers below ?

On Thu, Jul 9, 2015 at 7:05 AM, Guillaume Berche wrote:

Thanks Mike for the detailed notes.

It seems the toolsmiths-tracker url is wrong and pointing to the
buildpacks tracker, and the IF-tracker has not yet been configured for
being public (I get an "You are not allowed to view the requested page.")

[toolsmiths-tracker]: https://www.pivotaltracker.com/n/projects/1042066
[lf-tracker]: https://www.pivotaltracker.com/n/projects/939266

Thanks in advance,

Guillaume.

On Wed, Jul 8, 2015 at 4:58 PM, Mike Dalessio wrote:

Hi all,

We had a meeting of the Utilities PMC yesterday, permanent notes are at:


https://github.com/cloudfoundry/pmc-notes/blob/master/Utilities/2015-07-07-utilities.md

which I've helpfully copied into this email below.

Cheers,
-mike


-----

*# Utilities PMC Meeting 2015-07-07*

*## Agenda*

1. Update on CI tools (Mike Dalessio)
2. Update on CLI (Greg Oehmen)
3. Update on Eclipse plugin and Java tools (Ryan Morgan)
4. Proposal to accept receptor-client into incubation
5. Open Discussion


*## Attendees*

* Chip Childers, Cloud Foundry Foundation
* Mike Dalessio, Pivotal (PMC lead)
* Deepak Vij, Huawei


*## Update on CI tools (Mike Dalessio)*

- Krafa track of work has been broadly defined, and it's in the
Toolsmiths [public backlog and icebox][toolsmiths-tracker].

- A pair is continuing work on [License Finder][lf], which is an OSS tool
to
examine a project's declared dependencies (via a package manager)
and discover licenses and changes in licensing. This should be
useful for the Foundation as well. (See [Tracker here][lf-tracker].)


*## Update on CLI (Greg Oehman)*

The CLI team released version 6.12.1 on 7/6/2015. Release notes,
download instructions and binaries are available at
https://github.com/cloudfoundry/cli/releases


*## Update on Eclipse plugin and Java tools (Ryan Morgan)*

* Eclipse tooling 1.8.3 release (Nieraj posted notes about this to
cf-dev).


*## Proposal to accept receptor-client into incubation*

Last meeting, Ryan Morgan [proposed receptor-client for
acceptance][receptor-client-proposal] into incubation.



No objections having been raised, on either the mailing list or in the
PMC meeting, this is now accepted into incubation.


*## Open Discussion*

None.


-----

Links:

[toolsmiths-tracker]: https://www.pivotaltracker.com/n/projects/1042066
[lf-tracker]: https://www.pivotaltracker.com/n/projects/939266
[lf]: https://github.com/pivotal/LicenseFinder
[receptor-client-proposal]:
http://cf-dev.70369.x6.nabble.com/cf-dev-Proposal-for-incubation-of-receptor-client-Java-client-for-receptor-td440.html


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


App autosleep support

Guillaume Berche
 

Hi,

I wonder if there are plans to implement an auto-sleep behavior in
cloudfoundry, in which inactive apps would be automatically stopped after a
max inactivity threshold, and automatically restart upon arrival of traffic
on their routes. Similar to google app engine default behavior [1]

I did not find mentions of this yet in mailing lists and trackers.

We feel at Orange that such feature can improve the density for some of our
non-prod use-cases (with environmental and financial benefits).

I'd like to know if someone in the community already worked on such feature
or would be interested in collaborating on an opensource implementation.

I drafted some specs for a java-based implementation we're planning to work
on [2]. I'd love to hear feedbacks and suggestions on this.

Thanks in advance,

Guillaume.

[1] https://cloud.google.com/appengine/docs/java/modules/
[2]
https://docs.google.com/document/d/1tMhIBX3tw7kPEOMCzKhUgmtmr26GVxyXwUTwMO71THI/edit#


Re: Soliciting feedback on a UX change for route services

Guillaume Berche
 

Thanks Shannon for your feedback.

I understand there is a small window into which the pre-determined app
might not exist anymore (e.g. during blue/green deployment traffic shift).
The default behavior you're suggesting (picking a different app instance)
seems sensible to me, even though it will lead to seldom false associations.

We can imagine to refine this behavior in a second step, when use-cases of
router service being sensitive to false associations become more frequent:
allow the gorouter to comply to hints provided by the route service to tune
the behavior in case the pre-determined app might not exist anymore. The
router service could for instance augment the router-service HTTP header
with hints fields:


- missing-app-policy with one of the following values:
- reassign-app: the router transparently route the request to another
app(default)
- reject: reject the request (e.g. 502 status code with a json body
providing the currently available app ids). In this case the
route service
may reemit the request to the gorouter, specifying the second param below
- route-to-app override the predetermined app to which to route the
traffic


I'm currently planning to implement a route-service that would leverage the
app_id in the request in an "autosleep", see [1]. The "reassign-app"
default policy seems fine as a first step. The reject policy would be a
nice refinement to close this corner case.

[1]
https://docs.google.com/document/d/1tMhIBX3tw7kPEOMCzKhUgmtmr26GVxyXwUTwMO71THI/edit#

Guillaume.

On Sat, Jul 18, 2015 at 12:08 AM, Shannon Coen <scoen(a)pivotal.io> wrote:

Guillaume,

Including the app_id with the request forwarded to the route service
becomes misleading/false when, upon receiving the request back from the
route service, the pre-determined app no longer has instances available. At
that time GoRouter should be able to choose a different app instance for
the route, possibly of a different app, rather than rejecting the request
or re-forwarding the request to the route service with a different app id.
Otherwise, the route service may be making false associations.

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Thu, Jun 25, 2015 at 9:19 PM, Guillaume Berche <bercheg(a)gmail.com>
wrote:

I was about to suggest a similar UX for expressing a list of route
services, by relying on params ordering

cf update-route DOMAIN [-n HOST] (-s 'service instance' )*
cf update-route DOMAIN [-n HOST] -s caching -s https-only -s rate-limiting

Besides, If the MVP does not include support for multiple route services,
route service implementers might be able to experiment with supporting
arbitrary params as a way for users to specify fine grain options, possibly
ordered

cf create-service large-grain-route-service -p '{ "caching": true,
"ssl_only": true, "rate_limit": 3 }'

+1 for Mike's suggestions to allow for some route services to be
implemented in an upfront LB such as no router. This might address the
latency and availability concerns in the initial MVP ("route services to
forward requests back through the LB and GoRouter")

Lastly, it seems important that route services be able to output logs
that end up being associated with the app that receive the associated
traffic (e.g. cache hit or cache miss for a specific incoming request).
With route services being associated to routes (and not being bound to app
instances anymore), I'd like to re-iterate the suggestion I made in the
design document (on Feb 17) to have the gorouter include the app_id in the
headers of the signed request it sends to route service(s). This will allow
for a route service with log_emiter scope to add entries to the proper app
through loggregator/doppler. Of course, this also means that when a route
is associated to multiple apps, the load balancing decision among app is
made prior to sending traffic to route service(s). I'd imagine the app_id
could be propagated in the signed request headers when going through route
services and finally reaching the gorouter before hitting the app (as to
preserve the stateless nature of gorouter).

Guillaume.


On Fri, Jun 26, 2015 at 12:04 AM, Shannon Coen <scoen(a)pivotal.io> wrote:

This is great. Thank you, Mike.

FWIW, James had the following suggestion update-route could be used to
associate multiple routes, and express their chain order. We're not fixed
on this UX. We'll consider this more carefully when we get closer to the CF
CLI work.

cf update-route DOMAIN [-n HOST] [-s 'list,of,service,instances']

Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.

On Thu, Jun 25, 2015 at 12:58 PM, Mike Youngstrom <youngm(a)gmail.com>
wrote:


This is interesting. Could you flesh this out for me? What use cases
do you have in mind for associating a service instance with a route, but
not providing a forwarding address?

I would imagine you could bind a service to a route any time you want
to customize incoming traffic in some way. But that customization wouldn't
necessarily have to be implemented as a proxy.

Here are a few examples:

* A Public facing service as an indicator that a given route should be
made public facing. (Would require a broker to orchestrate stuff outside
of CF DNS, applying DoS security profiles to the route, force ssl on the
front end load balancer, etc.)
* A service to apply web front caching to a route. Could be done as a
proxy but could also be done by changing config in a front end load
balancer that supports caching like an F5 LTM.
* Rate limiting. Could be implemented as a proxy, or could be
implemented by applying some config in a front end load balancer
* A security service to limit client IP addresses allowed to connect on
a route. Again could be implemented as a proxy if you trust
X-Forwarded-For or simply change some config on a front end load balancer
no new proxy needed.

Basically a service applied to a route could trigger and manage all
kinds of functionality not necessarily implemented as proxy orchestrated by
the GoRouter.

It also occurs to me that the only time chain ordering of route
services seems to be an issue is in the case of a proxy url. So, it is
unfortunate that I'd be limited to binding only one route service when I
may want to apply all kinds of functionality to a route not implemented as
a proxy because user defined ordering isn't an issue.

That said I can see how it can be difficult for CF to provide a generic
solution to the kind of functionality applied above and that you may not
want to distract from the basic Route Services MVP to accommodate these
types of use cases. I guess I can only hope that you keep the concept of
applying non proxy functionality to a route in mind as you move through
your MVP.

Mike

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: CF-Abacus: incubation and inception meeting coming soon

Alexander Lomov <alexander.lomov@...>
 

Hello, Maxim.

I am Alexander Lomov. I work in R&D department at Altoros. I would like to
participate in your Hangout meeting dedicated to CF Abacus.

Could you please tell when and how I can participate.

Thank you,
Alex L.

------------------------
Alex Lomov
*Altoros* — Cloud Foundry deployment, training and integration
*Twitter:* @code1n <https://twitter.com/code1n> *GitHub:* @allomov
<https://gist.github.com/allomov>


Questions about /v2/app_usage_events

王小锋 <zzuwxf at gmail.com...>
 

Hi, there

I noticed there are three state about the app event, started, stopped and
buildpack_set.

what does "buildpack_set" mean? If the app is in buildpack_set status, does
it occupy cup/memory/disk?

If the app is deleted, will the event reflected in app_usage_events? if no,
should I use /v2/app_usage_events together with
/v2/events?q=type:audit.app.delete-request to determine the lifetime of an
app?

Many thanks.


Re: Removing FUSE support from CF

Jack Cai
 

+1 for space-level configuration.

Jack

On Wed, Jul 29, 2015 at 2:04 PM, Matt Cowger <matt(a)cowger.us> wrote:

We're wary of adding too many knobs to the platform and exposing them
all the way down to app developers increases the cognitive load for folks
using the platform. Enabling/disabling it on a per-installation level, and
- maybe - a per-space level, might be a decent compromise?

Agreed. I'd argue this this probably not be a 'per-app' thing, as I too
amy way of the knobs that developers like to turn. I think a per space
level is just the right level.

On Wed, Jul 29, 2015 at 10:20 AM, Onsi Fakhouri <ofakhouri(a)pivotal.io>
wrote:

That's still very much open for discussion. Obviously, someone with
administrative privileges should be in charge of this particular piece of
configuration.

Also making this a runtime config (e.g. feature flag) as opposed to a
deploy-time config (e.g. part of the CC config written out by BOSH) would
make the different behaviors more testable.

Thoughts? Preferences? We're wary of adding too many knobs to the
platform and exposing them all the way down to app developers increases the
cognitive load for folks using the platform. Enabling/disabling it on a
per-installation level, and - maybe - a per-space level, might be a decent
compromise?

Onsi



On Wed, Jul 29, 2015 at 9:54 AM, Matt Cowger <matt(a)cowger.us> wrote:

Once - configurable on a per-app, per space, or per deployment basis?

On Wed, Jul 29, 2015 at 9:50 AM, Onsi Fakhouri <ofakhouri(a)pivotal.io>
wrote:

Hey all,

Based on the feedback we got and the relatively low cost to maintain
privileged support we'd like to propose making running privileged
containers on the platform configurable - we will recommend this be turned
off when running untrusted workloads and it will likely default to off. We
have longer term plans to support mounting persistent volumes in Diego at
which point support for mounting FUSE in unprivileged containers can become
a reality.

Thoughts?

Onsi

On Mon, Jul 13, 2015 at 4:42 AM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Mon, Jul 13, 2015 at 2:48 AM, Lerenc, Vedran <vedran.lerenc(a)sap.com
wrote:
Hi Onsi,



Ø Thoughts? Concerns?



Well, that’s bad news.



We, and I assume many others as well (like the folks from Stackato
who do it in the public), have used SSHFS + FUSE to implement a persistent
file system for old-fashioned apps/apps that are not Cloud-native. I don’t
want to fight an ideological battle here, it’s just that these apps do
still exist (in majority) and a file system service is an important
service/feature for them.



So if you remove FUSE (which we thought is not going away/was added
to stay), it’s pretty bad for us/many apps.



Best regards, Vedran
+1 - It would be sad to see FUSE support go away. It's been very
helpful for running apps that depend on a persistent FS, like Wordpress.
Perhaps this use case of mounting a remote SSHFS could be supported in some
other way?

Dan








*From: *Onsi Fakhouri
*Reply-To: *"Discussions about Cloud Foundry projects and the system
overall."
*Date: *Saturday 11 July 2015 01:10
*To: *cf-dev
*Subject: *[cf-dev] Removing FUSE support from CF



Hey CF-Dev,



The Garden team has been hard at work substantially improving
Garden-Linux's security features. Garden-Linux now employs user namespaces
and drops capabilities when creating unprivileged containers - we're
excited to bring both of these features to the platform!



Diego currently runs applications in *privileged* containers. These
lack the security features outlined above and we are planning on switching
to launch all CF applications in *unprivileged* containers.



Unfortunately, it has proved difficult to support
mounting FUSE filesystems from within unprivileged containers. We believe
the security benefits outweigh the features that FUSE give us and* are
planning on removing support for FUSE in favor of better securing our
containers.* If/when FUSE support in unprivileged containers
becomes possible we may add it back to the platform.



Thoughts? Concerns?



Thanks!



Onsi

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Node.js Apps with small memory limits; Inaccurate Memory Availability in Containers

Christopher Piraino <cpiraino@...>
 

Sai,

Running free/top from inside the container is going to report the VM's
memory/cpu statistics in which the container is running. The correct stats
are located in the appropriate cgroups filesystem, which is where Cloud
Foundry pulls stats from when using the CLI.

From inside the container it is actually very hard to figure out how much
memory the cgroup is using (see this article for more information
<http://fabiokung.com/2014/03/13/memory-inside-linux-containers/>).

Best,
Chris

On Wed, Jul 29, 2015 at 7:55 AM, Mike Dalessio <mdalessio(a)pivotal.io> wrote:

Hi Sai,

Thanks for asking these questions. The buildpacks team, who currently
maintains the nodejs-buildpack, is totally open to improving the node.js
developer experience.

I'd love to hear about anyone's experience managing the total heap size
within the node.js interpreter. If you have played with this, let us know,
and we'd be happy to work with you on how it might work in conjunction with
container memory limits.

Cheers,
-mike


On Wed, Jul 29, 2015 at 10:47 AM, Sai Vennam <svennam92(a)gmail.com> wrote:

Hey All,

I've recently started investigating a memory issue with Node.js apps
running in CloudFoundry environments. FYI, I'm using CFv210. As an
example, if I push a Node.js app with a mem leak with a 512MB memory limit,
the Node.js V8 engine tries to allocate more and more memory until it
passes that memory limit and the application crashes. The behavior I expect
to see is that it will stop trying to allocate more memory when it reaches
the limit, and instead try to GC more aggressively (and then crash at a
later time).

By default, on 64 bit machines, the Node.js v8 engine has a 1GB heap
limit, so I can see why the engine tries to allocate more than is really
available. There should be some way to prevent the Node.js v8 engine from
trying to allocate more than is available. In Java, you can use JVM opts to
set heap limits, maybe something similar?

I did find one thing that might help, --max-old-space-size. But... has
any one done any investigation as to how to set that space size?
"--max-old-space-size" only accounts for the v8 engine's heap, not the
buffers or other processes. For example, should that limit be set to 50% of
the memory_limit? 75%? Maybe that's something the Node.js buildpack should
set as a reasonable default?

There is a separate issue that might be related to this. When you run
'free' or 'top' as a shell command from within the container spun up for my
application, I am seeing "32gb" total. This can't be right... I specified
512 when creating my application! When I run commands like "os.totalmem()"
from within Node.js, I'm also seeing 32gb.

There may be a better solution that doesn't involve setting any params,
but instead just fixing those kernel commands to be accurate.

Thanks,
Sai

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Notifications on ORG, SPACE and USER modifications

Piotr Przybylski <piotrp@...>
 

We have several use cases
- in time event processing for usage, with notification instead of polling
we'll be able to keep up better, getting closer to real time for metering,
and thus rating and billing
- timely display of the new system objects like spaces or organizations,
being notified about new objects, avoids retrieving all objects and
comparing which is already known
- detecting and signaling entity field or property changes, for example
name changes, which can require polling and comparison
These in turn would allow better support uses like
- console or UI reactive to entity changes
- activity dashboards
- analytics

Piotr



|------------>
| From: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Mike Youngstrom <youngm(a)gmail.com> |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| To: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|"Discussions about Cloud Foundry projects and the system overall." <cf-dev(a)lists.cloudfoundry.org> |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Date: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|07/29/2015 08:54 AM |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Subject: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|Re: [cf-dev] Notifications on ORG, SPACE and USER modifications |
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|------------>
| Sent by: |
|------------>
>--------------------------------------------------------------------------------------------------------------------------------------------------|
|cf-dev-bounces(a)lists.cloudfoundry.org |
>--------------------------------------------------------------------------------------------------------------------------------------------------|





For us the main use case is security auditing to keep a long term record of
who has done anything.  In the case of our Security team rather than use CF
events directly they preferred to have events forwarded to Security
Analytics.  Today we pull events then forward the details to Security
Analytics via a syslog endpoint.

Mike

[0] http://www.emc.com/security/security-analytics/security-analytics.htm

On Tue, Jul 28, 2015 at 11:26 PM, Dieu Cao <dcao(a)pivotal.io> wrote:
Hi all interested in notifications on modification of resources,

It would be helpful for me in framing the "what" and the "why" of this
feature if you could also describe your specific use cases and pain
points on why you would want notifications on modifications and also
which resources you particularly care about.
Is it for real time updates on a dashboard? For consumption for billing
purposes?  For triggering provisioning/deprovisioning of resources?

-Dieu

On Tue, Jul 28, 2015 at 11:05 AM, Jean-Sebastien Delfino <
jsdelfino(a)gmail.com> wrote:
I’m going to need something like this too for the CF Abacus service
metering project, as I’d like to track the lifecycle of orgs, users, etc
to match their history with the usage data reported for them.

Here’s a straw man description of what I had in mind:

- For Abacus, I’d need a Lossless API. Usage metering eventually
translates to billing and money, you don’t want to lose that :)

- An extension or variant of the current CF /v2/events API supporting
users, orgs, app usage etc, as even with a notification API I’ll still
need to do GETs sometimes.

- 304 responses with etags on these GETs (as suggested earlier in the
thread [1]) would be good.

- A Webhook style notification API where I could register interest in a
selection of events with a callback URL, and get these events POSTed
back to me at that URL, similar to what Github and many others do with
Webhooks.

- On top of Webhooks, it’d be nice to have a form of streaming (either
down to the client like the Firehose does or in the other direction up
to the Webhook callback URL), but I'm not sure if we’ll need that in the
project right away.

- We’d obviously need some form of security, maybe use my user token to
register for events on entities that I have access to?

- I’m also curious about the group’s thoughts on queueing and
back-pressure when events get generated faster that they can be consumed
for example. There was a mention of some message queuing earlier [2].
That would make sense to me (although IMO it’d be good if the underlying
MQ didn’t shine through the API). What did you have in mind for this?

I guess there are quite a few things to figure out here! I’ll be happy
to collaborate with the community on these discussions.

Thoughts?

[1]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p842.html
[2]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p834.html

- Jean-Sebastien



On Fri, Jul 24, 2015 at 9:59 PM, Matt Cowger <matt(a)cowger.us> wrote:
I think ETags is reasonable thought as well.

On Thu, Jul 23, 2015 at 4:39 PM, Benjamin Black <bblack(a)pivotal.io>
wrote:
ETags and a 304 response are specifically intended for that purpose.
I'd recommend that over relying on Last-Modified.


b

On Thu, Jul 23, 2015 at 12:34 AM, Koper, Dies <
diesk(a)fast.au.fujitsu.com> wrote:
Or setting the Last-Modified HTTP response header accordingly, and
allow clients to use HTTP caching mechanisms (Last-Modified, etc.)
to get quick empty responses with the current APIs if no changes
have been made? (Or maybe this is already working so – haven’t
checked).





Regards,


Dies Koper





From: cf-dev-bounces(a)lists.cloudfoundry.org [mailto:
cf-dev-bounces(a)lists.cloudfoundry.org] On Behalf Of Matt Cowger
Sent: Thursday, July 23, 2015 4:45 PM
To: Discussions about Cloud Foundry projects and the system
overall.


Subject: Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications





I've wanted something similar as well.





On a related note, having a CC API 'serial' number (for each object
in CC - apps, spaces, etc) that increments on every change relevant
to that object would be of value for detecting if something has
changed.





On Thu, Jul 23, 2015 at 3:27 PM, Dieu Cao <dcao(a)pivotal.io> wrote:


There are a few different approaches to this and different concerns
that are possible.


The requests I've seen have been around wanting to be able to
subscribe to and filter the various events that cc currently
generates so that other behavior could be triggered.


We currently have events, app usage events, and service usage
events.


Is it acceptable for the notifications to be lossy?  Depends on the
use case but If so, then the firehose may be an acceptable
approach.





The CAPI team is currently focusing on other work in the near term,
such as the v3 API and private brokers, but would be happy to
collaborate on a proposal.








On Wed, Jul 22, 2015 at 2:05 PM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:


My take:





CC should have callbacks on for each model create, update and
delete methods. Those callbacks will send a message to an MQ, which
you can subscribe to consume those messages.


This can be expanded to pretty much every event we need to track.


What do you think?





JP





2015-07-22 17:30 GMT-03:00 Matthias X Hub <matthias.hub(a)de.ibm.com
>:


Hi,

we (=IBM) are also having the need and are currently investigating
how to solve this. We plan to work on a proposal to discuss this
further with the cf community. I'll keep you updated on that.

Regards,
Matthias



From:        Mike Youngstrom <youngm(a)gmail.com>
To:        "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date:        22.07.2015 20:57
Subject:        Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications
Sent by:        cf-dev-bounces(a)lists.cloudfoundry.org








We have the same need.  Today we are polling the CC.

It would be nice for us also if we could get CC event notifications
via something like the firehose.

Mike

On Wed, Jul 22, 2015 at 10:23 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:
I mean, I know you can list those events thru the API, but I want
something that will react on an event instead of having to be
constantly polling for them.

2015-07-22 13:18 GMT-03:00 Juan Pablo Genovese <
juanpgenovese(a)gmail.com>:
Sree,

thanks! Any pointers on how can I hook up to these audit events?

Thank you!

2015-07-22 13:12 GMT-03:00 Sree Tummidi <stummidi(a)pivotal.io>:
I believe there are audit events generated for all these actions
which can be captured and forwarded to an SIEM solution like splunk


Thanks,
Sree

Sent from my iPhone

On Jul 22, 2015, at 8:54 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:

Guys,

I need to somehow hook up into the Cloud Controller (CC) to capture
ORG, SPACE and USER deletion, insertion and update.

So far, I considered some approaches, such as forking the CC (the
least favorite) and modifying the code with some hooks, tapping
into Nginx to capture the requests, and using triggers in the
database to capture each event and send the necessary info to a
service.

What do you think?
Any other idea you might have?

Thanks!

--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------


http://www.jpgenovese.com
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com



--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev









--


Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------


http://www.jpgenovese.com



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev






_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev









--


-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Jean-Sebastien

Sent from my DynaTAC 8000x

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: CF-Abacus: incubation and inception meeting coming soon

Stormy
 

I don't see anything on Twitter about this. Any objections to tweeting this
news?

Is there a potential blog post for the CF blog here?

Best,

Stormy


On Tue, Jul 28, 2015 at 11:15 PM, Michael Maximilien <maxim(a)us.ibm.com>
wrote:

Hi, all,

Now that CF-Abacus is officially an incubator under the guidance of the
CFF, here are some quick updates:

1. The project official github moved to:

https://github.com/cloudfoundry-incubator/cf-abacus

2. We are planning an inception next week Wednesday from 2p to 5p in SF.

We invite everyone interested to take a look at the repo, provide
feedback, or better, join us at the inception meeting. The location will be
either CFF, Pivotal, or IBM. All within a few blocks in downtown SF.

We will also have Google hangout and conference call for remote
participants.

If interested, then respond to me directly so I add you to the invite list.

Thanks and talk next week. Best,

CF-Abacus team

dr.max
ibm cloud labs
silicon valley, ca

Sent from my iPhone

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: CF-Abacus: incubation and inception meeting coming soon

Michael Maximilien
 

Quick update on inception meeting.

To accommodate our friends and colleagues from Europe who would like to attend, let's plan to move the meeting to 10a to 12:30p with the option of lunch after at nearby location in SF.

Unless I hear any objections I will send the invites to those interested parties who have already contacted me and confirm details here.

If you want to attend (local or remote) please remember to reply to me with email so I can add you to invite list.

Best,

dr.max
ibm cloud labs
silicon valley, ca

Sent from my iPhone

On Jul 28, 2015, at 10:15 PM, Michael Maximilien <maxim(a)us.ibm.com> wrote:

Hi, all,

Now that CF-Abacus is officially an incubator under the guidance of the CFF, here are some quick updates:

1. The project official github moved to:

https://github.com/cloudfoundry-incubator/cf-abacus

2. We are planning an inception next week Wednesday from 2p to 5p in SF.

We invite everyone interested to take a look at the repo, provide feedback, or better, join us at the inception meeting. The location will be either CFF, Pivotal, or IBM. All within a few blocks in downtown SF.

We will also have Google hangout and conference call for remote participants.

If interested, then respond to me directly so I add you to the invite list.

Thanks and talk next week. Best,

CF-Abacus team

dr.max
ibm cloud labs
silicon valley, ca

Sent from my iPhone


Re: Removing FUSE support from CF

Matt Cowger
 

We're wary of adding too many knobs to the platform and exposing them all
the way down to app developers increases the cognitive load for folks using
the platform. Enabling/disabling it on a per-installation level, and -
maybe - a per-space level, might be a decent compromise?

Agreed. I'd argue this this probably not be a 'per-app' thing, as I too
amy way of the knobs that developers like to turn. I think a per space
level is just the right level.

On Wed, Jul 29, 2015 at 10:20 AM, Onsi Fakhouri <ofakhouri(a)pivotal.io>
wrote:

That's still very much open for discussion. Obviously, someone with
administrative privileges should be in charge of this particular piece of
configuration.

Also making this a runtime config (e.g. feature flag) as opposed to a
deploy-time config (e.g. part of the CC config written out by BOSH) would
make the different behaviors more testable.

Thoughts? Preferences? We're wary of adding too many knobs to the
platform and exposing them all the way down to app developers increases the
cognitive load for folks using the platform. Enabling/disabling it on a
per-installation level, and - maybe - a per-space level, might be a decent
compromise?

Onsi



On Wed, Jul 29, 2015 at 9:54 AM, Matt Cowger <matt(a)cowger.us> wrote:

Once - configurable on a per-app, per space, or per deployment basis?

On Wed, Jul 29, 2015 at 9:50 AM, Onsi Fakhouri <ofakhouri(a)pivotal.io>
wrote:

Hey all,

Based on the feedback we got and the relatively low cost to maintain
privileged support we'd like to propose making running privileged
containers on the platform configurable - we will recommend this be turned
off when running untrusted workloads and it will likely default to off. We
have longer term plans to support mounting persistent volumes in Diego at
which point support for mounting FUSE in unprivileged containers can become
a reality.

Thoughts?

Onsi

On Mon, Jul 13, 2015 at 4:42 AM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Mon, Jul 13, 2015 at 2:48 AM, Lerenc, Vedran <vedran.lerenc(a)sap.com>
wrote:

Hi Onsi,



Ø Thoughts? Concerns?



Well, that’s bad news.



We, and I assume many others as well (like the folks from Stackato who
do it in the public), have used SSHFS + FUSE to implement a persistent file
system for old-fashioned apps/apps that are not Cloud-native. I don’t want
to fight an ideological battle here, it’s just that these apps do still
exist (in majority) and a file system service is an important
service/feature for them.



So if you remove FUSE (which we thought is not going away/was added to
stay), it’s pretty bad for us/many apps.



Best regards, Vedran
+1 - It would be sad to see FUSE support go away. It's been very
helpful for running apps that depend on a persistent FS, like Wordpress.
Perhaps this use case of mounting a remote SSHFS could be supported in some
other way?

Dan








*From: *Onsi Fakhouri
*Reply-To: *"Discussions about Cloud Foundry projects and the system
overall."
*Date: *Saturday 11 July 2015 01:10
*To: *cf-dev
*Subject: *[cf-dev] Removing FUSE support from CF



Hey CF-Dev,



The Garden team has been hard at work substantially improving
Garden-Linux's security features. Garden-Linux now employs user namespaces
and drops capabilities when creating unprivileged containers - we're
excited to bring both of these features to the platform!



Diego currently runs applications in *privileged* containers. These
lack the security features outlined above and we are planning on switching
to launch all CF applications in *unprivileged* containers.



Unfortunately, it has proved difficult to support
mounting FUSE filesystems from within unprivileged containers. We believe
the security benefits outweigh the features that FUSE give us and* are
planning on removing support for FUSE in favor of better securing our
containers.* If/when FUSE support in unprivileged containers becomes
possible we may add it back to the platform.



Thoughts? Concerns?



Thanks!



Onsi

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
-- Matt


Re: Removing FUSE support from CF (Onsi Fakhouri)

Onsi Fakhouri <ofakhouri@...>
 

Hi Pramod,

Still TBD - we're driving hard for a Diego GA first and will then evaluate
the various priorities and scope them out.

Onsi

On Wed, Jul 29, 2015 at 9:59 AM, Pramod Mandagere <nagapramod(a)gmail.com>
wrote:

If not FUSE, what is the timeline/status for support of mounting
persistent volumes in Diego natively(remember seeing a talk in cf summit on
floating/fixed volumes)?
Pramod


On Wed, Jul 29, 2015 at 9:53 AM <cf-dev-request(a)lists.cloudfoundry.org>
wrote:

Send cf-dev mailing list submissions to
cf-dev(a)lists.cloudfoundry.org

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
or, via email, send a message with subject or body 'help' to
cf-dev-request(a)lists.cloudfoundry.org

You can reach the person managing the list at
cf-dev-owner(a)lists.cloudfoundry.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of cf-dev digest..."


Today's Topics:

1. Re: Removing FUSE support from CF (Onsi Fakhouri)
2. Droplets and Stacks (Colin Humphreys)
3. Re: Notifications on ORG, SPACE and USER modifications
(Matt Cowger)


----------------------------------------------------------------------

Message: 1
Date: Wed, 29 Jul 2015 09:50:00 -0700
From: Onsi Fakhouri <ofakhouri(a)pivotal.io>
To: "Discussions about Cloud Foundry projects and the system overall."
<cf-dev(a)lists.cloudfoundry.org>
Subject: Re: [cf-dev] Removing FUSE support from CF
Message-ID:
<CAFwdB-zsTVY4-KUzUmUOiHZ7XrWhvSzc=rnOhmXmkfBq=
YCFqQ(a)mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hey all,

Based on the feedback we got and the relatively low cost to maintain
privileged support we'd like to propose making running privileged
containers on the platform configurable - we will recommend this be turned
off when running untrusted workloads and it will likely default to off.
We
have longer term plans to support mounting persistent volumes in Diego at
which point support for mounting FUSE in unprivileged containers can
become
a reality.

Thoughts?

Onsi

On Mon, Jul 13, 2015 at 4:42 AM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Mon, Jul 13, 2015 at 2:48 AM, Lerenc, Vedran <vedran.lerenc(a)sap.com>
wrote:

Hi Onsi,



? Thoughts? Concerns?



Well, that?s bad news.



We, and I assume many others as well (like the folks from Stackato who
do
it in the public), have used SSHFS + FUSE to implement a persistent
file
system for old-fashioned apps/apps that are not Cloud-native. I don?t
want
to fight an ideological battle here, it?s just that these apps do still
exist (in majority) and a file system service is an important
service/feature for them.



So if you remove FUSE (which we thought is not going away/was added to
stay), it?s pretty bad for us/many apps.



Best regards, Vedran
+1 - It would be sad to see FUSE support go away. It's been very
helpful
for running apps that depend on a persistent FS, like Wordpress.
Perhaps
this use case of mounting a remote SSHFS could be supported in some
other
way?

Dan








*From: *Onsi Fakhouri
*Reply-To: *"Discussions about Cloud Foundry projects and the system
overall."
*Date: *Saturday 11 July 2015 01:10
*To: *cf-dev
*Subject: *[cf-dev] Removing FUSE support from CF



Hey CF-Dev,



The Garden team has been hard at work substantially improving
Garden-Linux's security features. Garden-Linux now employs user
namespaces
and drops capabilities when creating unprivileged containers - we're
excited to bring both of these features to the platform!



Diego currently runs applications in *privileged* containers. These
lack the security features outlined above and we are planning on
switching
to launch all CF applications in *unprivileged* containers.



Unfortunately, it has proved difficult to support
mounting FUSE filesystems from within unprivileged containers. We
believe
the security benefits outweigh the features that FUSE give us and* are
planning on removing support for FUSE in favor of better securing our
containers.* If/when FUSE support in unprivileged containers becomes
possible we may add it back to the platform.



Thoughts? Concerns?



Thanks!



Onsi

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://lists.cloudfoundry.org/pipermail/cf-dev/attachments/20150729/94321642/attachment-0001.html
------------------------------

Message: 2
Date: Wed, 29 Jul 2015 17:51:42 +0100
From: Colin Humphreys <colin(a)cloudcredo.com>
To: cf-dev(a)lists.cloudfoundry.org
Subject: [cf-dev] Droplets and Stacks
Message-ID:
<CACfkAi+54GWe8c1pO8uDxWs_Sa4JjBkTQ5Ed+Xd8bLZRJWvg=
A(a)mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi All,

I wrote a couple of articles about droplets and stacks.

http://www.cloudcredo.com/a-droplet-of-value/

http://www.cloudcredo.com/stacks-of-problems/

The droplet post is fairly self-explanatory, and enabling the choice of
shipping droplets or source is well on the way in Cloud Foundry
development.

I feel our story around stacks is far less complete. It seems to be an
overloaded concept inherited from Heroku and the contract with the stack
seems to cause issues for both app and buildpack developers.

I'd like to open the discussion on what the future should be for stacks,
or
if you think they're perfect as they are.

Cheers,
Colin

CloudCredo Cheerleader
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://lists.cloudfoundry.org/pipermail/cf-dev/attachments/20150729/d774b045/attachment-0001.html
------------------------------

Message: 3
Date: Wed, 29 Jul 2015 09:52:53 -0700
From: Matt Cowger <matt(a)cowger.us>
To: "Discussions about Cloud Foundry projects and the system overall."
<cf-dev(a)lists.cloudfoundry.org>
Subject: Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications
Message-ID:
<
CAOuA4vytsHYam7YydbL+eiW1EGByXnoT+QPKD2NtoMQexzZmfQ(a)mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

For me the primary use case to developing management tools on top of the
CC
API, and being able to intelligently cache information about the state of
those resources. Without etags or serial numbers or similar, that means
every refresh needs to query a TON of data from CC.



On Tue, Jul 28, 2015 at 10:26 PM, Dieu Cao <dcao(a)pivotal.io> wrote:

Hi all interested in notifications on modification of resources,

It would be helpful for me in framing the "what" and the "why" of this
feature if you could also describe your specific use cases and pain
points
on why you would want notifications on modifications and also which
resources you particularly care about.
Is it for real time updates on a dashboard? For consumption for billing
purposes? For triggering provisioning/deprovisioning of resources?

-Dieu

On Tue, Jul 28, 2015 at 11:05 AM, Jean-Sebastien Delfino <
jsdelfino(a)gmail.com> wrote:

I?m going to need something like this too for the CF Abacus service
metering project, as I?d like to track the lifecycle of orgs, users,
etc to
match their history with the usage data reported for them.


Here?s a straw man description of what I had in mind:


- For Abacus, I?d need a Lossless API. Usage metering eventually
translates to billing and money, you don?t want to lose that :)


- An extension or variant of the current CF /v2/events API supporting
users, orgs, app usage etc, as even with a notification API I?ll still
need
to do GETs sometimes.


- 304 responses with etags on these GETs (as suggested earlier in the
thread [1]) would be good.


- A Webhook style notification API where I could register interest in a
selection of events with a callback URL, and get these events POSTed
back
to me at that URL, similar to what Github and many others do with
Webhooks.


- On top of Webhooks, it?d be nice to have a form of streaming (either
down to the client like the Firehose does or in the other direction up
to
the Webhook callback URL), but I'm not sure if we?ll need that in the
project right away.


- We?d obviously need some form of security, maybe use my user token to
register for events on entities that I have access to?


- I?m also curious about the group?s thoughts on queueing and
back-pressure when events get generated faster that they can be
consumed
for example. There was a mention of some message queuing earlier [2].
That
would make sense to me (although IMO it?d be good if the underlying MQ
didn?t shine through the API). What did you have in mind for this?


I guess there are quite a few things to figure out here! I?ll be happy
to
collaborate with the community on these discussions.


Thoughts?


[1]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p842.html

[2]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p834.html


- Jean-Sebastien

On Fri, Jul 24, 2015 at 9:59 PM, Matt Cowger <matt(a)cowger.us> wrote:

I think ETags is reasonable thought as well.

On Thu, Jul 23, 2015 at 4:39 PM, Benjamin Black <bblack(a)pivotal.io>
wrote:

ETags and a 304 response are specifically intended for that purpose.
I'd recommend that over relying on Last-Modified.


b

On Thu, Jul 23, 2015 at 12:34 AM, Koper, Dies <
diesk(a)fast.au.fujitsu.com> wrote:

Or setting the Last-Modified HTTP response header accordingly, and
allow clients to use HTTP caching mechanisms (Last-Modified, etc.)
to get
quick empty responses with the current APIs if no changes have been
made?
(Or maybe this is already working so ? haven?t checked).



Regards,

Dies Koper



*From:* cf-dev-bounces(a)lists.cloudfoundry.org [mailto:
cf-dev-bounces(a)lists.cloudfoundry.org] *On Behalf Of *Matt Cowger
*Sent:* Thursday, July 23, 2015 4:45 PM
*To:* Discussions about Cloud Foundry projects and the system
overall.
*Subject:* Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications



I've wanted something similar as well.



On a related note, having a CC API 'serial' number (for each object
in
CC - apps, spaces, etc) that increments on every change relevant to
that
object would be of value for detecting if something has changed.



On Thu, Jul 23, 2015 at 3:27 PM, Dieu Cao <dcao(a)pivotal.io> wrote:

There are a few different approaches to this and different concerns
that are possible.

The requests I've seen have been around wanting to be able to
subscribe to and filter the various events that cc currently
generates so
that other behavior could be triggered.

We currently have events, app usage events, and service usage
events.

Is it acceptable for the notifications to be lossy? Depends on the
use case but If so, then the firehose may be an acceptable approach.



The CAPI team is currently focusing on other work in the near term,
such as the v3 API and private brokers, but would be happy to
collaborate
on a proposal.





On Wed, Jul 22, 2015 at 2:05 PM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:

My take:



CC should have callbacks on for each model create, update and delete
methods. Those callbacks will send a message to an MQ, which you can
subscribe to consume those messages.

This can be expanded to pretty much every event we need to track.

What do you think?



JP



2015-07-22 17:30 GMT-03:00 Matthias X Hub <matthias.hub(a)de.ibm.com
:

Hi,

we (=IBM) are also having the need and are currently investigating
how
to solve this. We plan to work on a proposal to discuss this
further with
the cf community. I'll keep you updated on that.

Regards,
Matthias



From: Mike Youngstrom <youngm(a)gmail.com>
To: "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date: 22.07.2015 20:57
Subject: Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications
Sent by: cf-dev-bounces(a)lists.cloudfoundry.org
------------------------------




We have the same need. Today we are polling the CC.

It would be nice for us also if we could get CC event notifications
via something like the firehose.

Mike

On Wed, Jul 22, 2015 at 10:23 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:
I mean, I know you can list those events thru the API, but I want
something that will react on an event instead of having to be
constantly
polling for them.

2015-07-22 13:18 GMT-03:00 Juan Pablo Genovese <
juanpgenovese(a)gmail.com>:
Sree,

thanks! Any pointers on how can I hook up to these audit events?

Thank you!

2015-07-22 13:12 GMT-03:00 Sree Tummidi <stummidi(a)pivotal.io>:
I believe there are audit events generated for all these actions
which
can be captured and forwarded to an SIEM solution like splunk

Thanks,
Sree

Sent from my iPhone

On Jul 22, 2015, at 8:54 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:

Guys,

I need to somehow hook up into the Cloud Controller (CC) to capture
ORG, SPACE and USER deletion, insertion and update.

So far, I considered some approaches, such as forking the CC (the
least favorite) and modifying the code with some hooks, tapping
into Nginx
to capture the requests, and using triggers in the database to
capture each
event and send the necessary info to a service.

What do you think?
Any other idea you might have?

Thanks!

--
Mis mejores deseos,
Best wishes,
Meilleurs v?ux,

Juan Pablo
------------------------------------------------------

http://www.jpgenovese.com
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Mis mejores deseos,
Best wishes,
Meilleurs v?ux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com



--
Mis mejores deseos,
Best wishes,
Meilleurs v?ux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev





--

Mis mejores deseos,
Best wishes,
Meilleurs v?ux,

Juan Pablo
------------------------------------------------------

http://www.jpgenovese.com


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev





--

-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Jean-Sebastien

Sent from my DynaTAC 8000x

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
-- Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://lists.cloudfoundry.org/pipermail/cf-dev/attachments/20150729/7488b9a8/attachment.html
------------------------------

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


End of cf-dev Digest, Vol 4, Issue 126
**************************************
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Removing FUSE support from CF

Onsi Fakhouri <ofakhouri@...>
 

That's still very much open for discussion. Obviously, someone with
administrative privileges should be in charge of this particular piece of
configuration.

Also making this a runtime config (e.g. feature flag) as opposed to a
deploy-time config (e.g. part of the CC config written out by BOSH) would
make the different behaviors more testable.

Thoughts? Preferences? We're wary of adding too many knobs to the
platform and exposing them all the way down to app developers increases the
cognitive load for folks using the platform. Enabling/disabling it on a
per-installation level, and - maybe - a per-space level, might be a decent
compromise?

Onsi

On Wed, Jul 29, 2015 at 9:54 AM, Matt Cowger <matt(a)cowger.us> wrote:

Once - configurable on a per-app, per space, or per deployment basis?

On Wed, Jul 29, 2015 at 9:50 AM, Onsi Fakhouri <ofakhouri(a)pivotal.io>
wrote:

Hey all,

Based on the feedback we got and the relatively low cost to maintain
privileged support we'd like to propose making running privileged
containers on the platform configurable - we will recommend this be turned
off when running untrusted workloads and it will likely default to off. We
have longer term plans to support mounting persistent volumes in Diego at
which point support for mounting FUSE in unprivileged containers can become
a reality.

Thoughts?

Onsi

On Mon, Jul 13, 2015 at 4:42 AM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Mon, Jul 13, 2015 at 2:48 AM, Lerenc, Vedran <vedran.lerenc(a)sap.com>
wrote:

Hi Onsi,



Ø Thoughts? Concerns?



Well, that’s bad news.



We, and I assume many others as well (like the folks from Stackato who
do it in the public), have used SSHFS + FUSE to implement a persistent file
system for old-fashioned apps/apps that are not Cloud-native. I don’t want
to fight an ideological battle here, it’s just that these apps do still
exist (in majority) and a file system service is an important
service/feature for them.



So if you remove FUSE (which we thought is not going away/was added to
stay), it’s pretty bad for us/many apps.



Best regards, Vedran
+1 - It would be sad to see FUSE support go away. It's been very
helpful for running apps that depend on a persistent FS, like Wordpress.
Perhaps this use case of mounting a remote SSHFS could be supported in some
other way?

Dan








*From: *Onsi Fakhouri
*Reply-To: *"Discussions about Cloud Foundry projects and the system
overall."
*Date: *Saturday 11 July 2015 01:10
*To: *cf-dev
*Subject: *[cf-dev] Removing FUSE support from CF



Hey CF-Dev,



The Garden team has been hard at work substantially improving
Garden-Linux's security features. Garden-Linux now employs user namespaces
and drops capabilities when creating unprivileged containers - we're
excited to bring both of these features to the platform!



Diego currently runs applications in *privileged* containers. These
lack the security features outlined above and we are planning on switching
to launch all CF applications in *unprivileged* containers.



Unfortunately, it has proved difficult to support
mounting FUSE filesystems from within unprivileged containers. We believe
the security benefits outweigh the features that FUSE give us and* are
planning on removing support for FUSE in favor of better securing our
containers.* If/when FUSE support in unprivileged containers becomes
possible we may add it back to the platform.



Thoughts? Concerns?



Thanks!



Onsi

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Droplets and Stacks

Onsi Fakhouri <ofakhouri@...>
 

Hey Colin,

Good stuff. I like to draw a circle around the rootfs, the buildpacks, the
generated droplet, the Task/LRP recipes, and the lifecycle binaries that
run inside containers to stage and launch droplets. You could label that
circle an application lifecycle. Diego currently supports three
application lifecycles and is loosely coupled to those lifecycles:

1. The Linux-Buildpack App lifecycle: includes the cflinuxfs2 rootfs, the
various buildpacks (including a known interface for building custom
buildpacks), the droplets (compiled artifacts guaranteed to run with
cflinuxfs2), two binaries: the builder (performs staging) and the launcher
(runs applications), and code that can convert CC requests for staging and
running instances to Diego Task/LRP recipes.

2. The Windows App lifecycle: includes the notion of a correctly configured
windows environment, a windows-compatible droplet, a builder, a launcher,
and code that can generate Tasks/LRPs. In this context we do not yet
have/need the notion of a buildpack though we are free to add one later.
The builder simply prepares the droplet from source and the launcher knows
how to invoke it.

3. The Docker App lifecycle: has no rootfs as the docker image provides the
entire rootfs, includes a builder to extract docker-metadata and send it
back to CC for safe-keeping, and a launcher to launch the requested process
*and* present it with a standard CF environment. Again, there's also code
that knows how to translate CC requests for a docker-based application into
Tasks and LRPs.

The cool thing is that Diego doesn't care about any of these details and
you are free to construct your own lifecycles and have your own contracts
within each lifecycle. You are spot on in noting that there is an implicit
contract between the buildpacks and the rootfs. I'd go further and say
that that implicit contract covers everything in the lifecycle circle (e.g.
the builder has a contract with the buildpacks, it expects `detect`,
`compile` and `release` to work a certain way, the recipes have a contract
with the builder/launcher, they expect particular command line arguments,
etc...)

This is one reason why we've recently transitioned the ownership of the
rootfs from the runtime team to the buildpack team - as the buildpack team
is best suited to define and maintain the contract between the buildpacks
and the rootfs. Would love to explore ways to make all these contracts
more explicit.

One last point. I didn't use the word "stack" in this e-mail until just
now. I agree that it's an overloaded concept that is easily and often
misunderstood ;)

Onsi

On Wed, Jul 29, 2015 at 9:51 AM, Colin Humphreys <colin(a)cloudcredo.com>
wrote:

Hi All,

I wrote a couple of articles about droplets and stacks.

http://www.cloudcredo.com/a-droplet-of-value/

http://www.cloudcredo.com/stacks-of-problems/

The droplet post is fairly self-explanatory, and enabling the choice of
shipping droplets or source is well on the way in Cloud Foundry development.

I feel our story around stacks is far less complete. It seems to be an
overloaded concept inherited from Heroku and the contract with the stack
seems to cause issues for both app and buildpack developers.

I'd like to open the discussion on what the future should be for stacks,
or if you think they're perfect as they are.

Cheers,
Colin

CloudCredo Cheerleader

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Removing FUSE support from CF (Onsi Fakhouri)

Pramod Mandagere
 

If not FUSE, what is the timeline/status for support of mounting persistent
volumes in Diego natively(remember seeing a talk in cf summit on
floating/fixed volumes)?
Pramod


On Wed, Jul 29, 2015 at 9:53 AM <cf-dev-request(a)lists.cloudfoundry.org>
wrote:

Send cf-dev mailing list submissions to
cf-dev(a)lists.cloudfoundry.org

To subscribe or unsubscribe via the World Wide Web, visit
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
or, via email, send a message with subject or body 'help' to
cf-dev-request(a)lists.cloudfoundry.org

You can reach the person managing the list at
cf-dev-owner(a)lists.cloudfoundry.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of cf-dev digest..."


Today's Topics:

1. Re: Removing FUSE support from CF (Onsi Fakhouri)
2. Droplets and Stacks (Colin Humphreys)
3. Re: Notifications on ORG, SPACE and USER modifications
(Matt Cowger)


----------------------------------------------------------------------

Message: 1
Date: Wed, 29 Jul 2015 09:50:00 -0700
From: Onsi Fakhouri <ofakhouri(a)pivotal.io>
To: "Discussions about Cloud Foundry projects and the system overall."
<cf-dev(a)lists.cloudfoundry.org>
Subject: Re: [cf-dev] Removing FUSE support from CF
Message-ID:
<CAFwdB-zsTVY4-KUzUmUOiHZ7XrWhvSzc=rnOhmXmkfBq=
YCFqQ(a)mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hey all,

Based on the feedback we got and the relatively low cost to maintain
privileged support we'd like to propose making running privileged
containers on the platform configurable - we will recommend this be turned
off when running untrusted workloads and it will likely default to off. We
have longer term plans to support mounting persistent volumes in Diego at
which point support for mounting FUSE in unprivileged containers can become
a reality.

Thoughts?

Onsi

On Mon, Jul 13, 2015 at 4:42 AM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

On Mon, Jul 13, 2015 at 2:48 AM, Lerenc, Vedran <vedran.lerenc(a)sap.com>
wrote:

Hi Onsi,



? Thoughts? Concerns?



Well, that?s bad news.



We, and I assume many others as well (like the folks from Stackato who
do
it in the public), have used SSHFS + FUSE to implement a persistent file
system for old-fashioned apps/apps that are not Cloud-native. I don?t
want
to fight an ideological battle here, it?s just that these apps do still
exist (in majority) and a file system service is an important
service/feature for them.



So if you remove FUSE (which we thought is not going away/was added to
stay), it?s pretty bad for us/many apps.



Best regards, Vedran
+1 - It would be sad to see FUSE support go away. It's been very helpful
for running apps that depend on a persistent FS, like Wordpress. Perhaps
this use case of mounting a remote SSHFS could be supported in some other
way?

Dan








*From: *Onsi Fakhouri
*Reply-To: *"Discussions about Cloud Foundry projects and the system
overall."
*Date: *Saturday 11 July 2015 01:10
*To: *cf-dev
*Subject: *[cf-dev] Removing FUSE support from CF



Hey CF-Dev,



The Garden team has been hard at work substantially improving
Garden-Linux's security features. Garden-Linux now employs user
namespaces
and drops capabilities when creating unprivileged containers - we're
excited to bring both of these features to the platform!



Diego currently runs applications in *privileged* containers. These
lack the security features outlined above and we are planning on
switching
to launch all CF applications in *unprivileged* containers.



Unfortunately, it has proved difficult to support
mounting FUSE filesystems from within unprivileged containers. We
believe
the security benefits outweigh the features that FUSE give us and* are
planning on removing support for FUSE in favor of better securing our
containers.* If/when FUSE support in unprivileged containers becomes
possible we may add it back to the platform.



Thoughts? Concerns?



Thanks!



Onsi

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://lists.cloudfoundry.org/pipermail/cf-dev/attachments/20150729/94321642/attachment-0001.html
------------------------------

Message: 2
Date: Wed, 29 Jul 2015 17:51:42 +0100
From: Colin Humphreys <colin(a)cloudcredo.com>
To: cf-dev(a)lists.cloudfoundry.org
Subject: [cf-dev] Droplets and Stacks
Message-ID:
<CACfkAi+54GWe8c1pO8uDxWs_Sa4JjBkTQ5Ed+Xd8bLZRJWvg=
A(a)mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi All,

I wrote a couple of articles about droplets and stacks.

http://www.cloudcredo.com/a-droplet-of-value/

http://www.cloudcredo.com/stacks-of-problems/

The droplet post is fairly self-explanatory, and enabling the choice of
shipping droplets or source is well on the way in Cloud Foundry
development.

I feel our story around stacks is far less complete. It seems to be an
overloaded concept inherited from Heroku and the contract with the stack
seems to cause issues for both app and buildpack developers.

I'd like to open the discussion on what the future should be for stacks, or
if you think they're perfect as they are.

Cheers,
Colin

CloudCredo Cheerleader
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://lists.cloudfoundry.org/pipermail/cf-dev/attachments/20150729/d774b045/attachment-0001.html
------------------------------

Message: 3
Date: Wed, 29 Jul 2015 09:52:53 -0700
From: Matt Cowger <matt(a)cowger.us>
To: "Discussions about Cloud Foundry projects and the system overall."
<cf-dev(a)lists.cloudfoundry.org>
Subject: Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications
Message-ID:
<
CAOuA4vytsHYam7YydbL+eiW1EGByXnoT+QPKD2NtoMQexzZmfQ(a)mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

For me the primary use case to developing management tools on top of the CC
API, and being able to intelligently cache information about the state of
those resources. Without etags or serial numbers or similar, that means
every refresh needs to query a TON of data from CC.



On Tue, Jul 28, 2015 at 10:26 PM, Dieu Cao <dcao(a)pivotal.io> wrote:

Hi all interested in notifications on modification of resources,

It would be helpful for me in framing the "what" and the "why" of this
feature if you could also describe your specific use cases and pain
points
on why you would want notifications on modifications and also which
resources you particularly care about.
Is it for real time updates on a dashboard? For consumption for billing
purposes? For triggering provisioning/deprovisioning of resources?

-Dieu

On Tue, Jul 28, 2015 at 11:05 AM, Jean-Sebastien Delfino <
jsdelfino(a)gmail.com> wrote:

I?m going to need something like this too for the CF Abacus service
metering project, as I?d like to track the lifecycle of orgs, users,
etc to
match their history with the usage data reported for them.


Here?s a straw man description of what I had in mind:


- For Abacus, I?d need a Lossless API. Usage metering eventually
translates to billing and money, you don?t want to lose that :)


- An extension or variant of the current CF /v2/events API supporting
users, orgs, app usage etc, as even with a notification API I?ll still
need
to do GETs sometimes.


- 304 responses with etags on these GETs (as suggested earlier in the
thread [1]) would be good.


- A Webhook style notification API where I could register interest in a
selection of events with a callback URL, and get these events POSTed
back
to me at that URL, similar to what Github and many others do with
Webhooks.


- On top of Webhooks, it?d be nice to have a form of streaming (either
down to the client like the Firehose does or in the other direction up
to
the Webhook callback URL), but I'm not sure if we?ll need that in the
project right away.


- We?d obviously need some form of security, maybe use my user token to
register for events on entities that I have access to?


- I?m also curious about the group?s thoughts on queueing and
back-pressure when events get generated faster that they can be consumed
for example. There was a mention of some message queuing earlier [2].
That
would make sense to me (although IMO it?d be good if the underlying MQ
didn?t shine through the API). What did you have in mind for this?


I guess there are quite a few things to figure out here! I?ll be happy
to
collaborate with the community on these discussions.


Thoughts?


[1]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p842.html

[2]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p834.html


- Jean-Sebastien

On Fri, Jul 24, 2015 at 9:59 PM, Matt Cowger <matt(a)cowger.us> wrote:

I think ETags is reasonable thought as well.

On Thu, Jul 23, 2015 at 4:39 PM, Benjamin Black <bblack(a)pivotal.io>
wrote:

ETags and a 304 response are specifically intended for that purpose.
I'd recommend that over relying on Last-Modified.


b

On Thu, Jul 23, 2015 at 12:34 AM, Koper, Dies <
diesk(a)fast.au.fujitsu.com> wrote:

Or setting the Last-Modified HTTP response header accordingly, and
allow clients to use HTTP caching mechanisms (Last-Modified, etc.)
to get
quick empty responses with the current APIs if no changes have been
made?
(Or maybe this is already working so ? haven?t checked).



Regards,

Dies Koper



*From:* cf-dev-bounces(a)lists.cloudfoundry.org [mailto:
cf-dev-bounces(a)lists.cloudfoundry.org] *On Behalf Of *Matt Cowger
*Sent:* Thursday, July 23, 2015 4:45 PM
*To:* Discussions about Cloud Foundry projects and the system
overall.
*Subject:* Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications



I've wanted something similar as well.



On a related note, having a CC API 'serial' number (for each object
in
CC - apps, spaces, etc) that increments on every change relevant to
that
object would be of value for detecting if something has changed.



On Thu, Jul 23, 2015 at 3:27 PM, Dieu Cao <dcao(a)pivotal.io> wrote:

There are a few different approaches to this and different concerns
that are possible.

The requests I've seen have been around wanting to be able to
subscribe to and filter the various events that cc currently
generates so
that other behavior could be triggered.

We currently have events, app usage events, and service usage events.

Is it acceptable for the notifications to be lossy? Depends on the
use case but If so, then the firehose may be an acceptable approach.



The CAPI team is currently focusing on other work in the near term,
such as the v3 API and private brokers, but would be happy to
collaborate
on a proposal.





On Wed, Jul 22, 2015 at 2:05 PM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:

My take:



CC should have callbacks on for each model create, update and delete
methods. Those callbacks will send a message to an MQ, which you can
subscribe to consume those messages.

This can be expanded to pretty much every event we need to track.

What do you think?



JP



2015-07-22 17:30 GMT-03:00 Matthias X Hub <matthias.hub(a)de.ibm.com>:

Hi,

we (=IBM) are also having the need and are currently investigating
how
to solve this. We plan to work on a proposal to discuss this further
with
the cf community. I'll keep you updated on that.

Regards,
Matthias



From: Mike Youngstrom <youngm(a)gmail.com>
To: "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date: 22.07.2015 20:57
Subject: Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications
Sent by: cf-dev-bounces(a)lists.cloudfoundry.org
------------------------------




We have the same need. Today we are polling the CC.

It would be nice for us also if we could get CC event notifications
via something like the firehose.

Mike

On Wed, Jul 22, 2015 at 10:23 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:
I mean, I know you can list those events thru the API, but I want
something that will react on an event instead of having to be
constantly
polling for them.

2015-07-22 13:18 GMT-03:00 Juan Pablo Genovese <
juanpgenovese(a)gmail.com>:
Sree,

thanks! Any pointers on how can I hook up to these audit events?

Thank you!

2015-07-22 13:12 GMT-03:00 Sree Tummidi <stummidi(a)pivotal.io>:
I believe there are audit events generated for all these actions
which
can be captured and forwarded to an SIEM solution like splunk

Thanks,
Sree

Sent from my iPhone

On Jul 22, 2015, at 8:54 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:

Guys,

I need to somehow hook up into the Cloud Controller (CC) to capture
ORG, SPACE and USER deletion, insertion and update.

So far, I considered some approaches, such as forking the CC (the
least favorite) and modifying the code with some hooks, tapping into
Nginx
to capture the requests, and using triggers in the database to
capture each
event and send the necessary info to a service.

What do you think?
Any other idea you might have?

Thanks!

--
Mis mejores deseos,
Best wishes,
Meilleurs v?ux,

Juan Pablo
------------------------------------------------------

http://www.jpgenovese.com
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Mis mejores deseos,
Best wishes,
Meilleurs v?ux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com



--
Mis mejores deseos,
Best wishes,
Meilleurs v?ux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev





--

Mis mejores deseos,
Best wishes,
Meilleurs v?ux,

Juan Pablo
------------------------------------------------------

http://www.jpgenovese.com


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev





--

-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Jean-Sebastien

Sent from my DynaTAC 8000x

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
-- Matt
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://lists.cloudfoundry.org/pipermail/cf-dev/attachments/20150729/7488b9a8/attachment.html
------------------------------

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


End of cf-dev Digest, Vol 4, Issue 126
**************************************


Re: Removing FUSE support from CF

Matt Cowger
 

Once - configurable on a per-app, per space, or per deployment basis?

On Wed, Jul 29, 2015 at 9:50 AM, Onsi Fakhouri <ofakhouri(a)pivotal.io> wrote:

Hey all,

Based on the feedback we got and the relatively low cost to maintain
privileged support we'd like to propose making running privileged
containers on the platform configurable - we will recommend this be turned
off when running untrusted workloads and it will likely default to off. We
have longer term plans to support mounting persistent volumes in Diego at
which point support for mounting FUSE in unprivileged containers can become
a reality.

Thoughts?

Onsi

On Mon, Jul 13, 2015 at 4:42 AM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

On Mon, Jul 13, 2015 at 2:48 AM, Lerenc, Vedran <vedran.lerenc(a)sap.com>
wrote:

Hi Onsi,



Ø Thoughts? Concerns?



Well, that’s bad news.



We, and I assume many others as well (like the folks from Stackato who
do it in the public), have used SSHFS + FUSE to implement a persistent file
system for old-fashioned apps/apps that are not Cloud-native. I don’t want
to fight an ideological battle here, it’s just that these apps do still
exist (in majority) and a file system service is an important
service/feature for them.



So if you remove FUSE (which we thought is not going away/was added to
stay), it’s pretty bad for us/many apps.



Best regards, Vedran
+1 - It would be sad to see FUSE support go away. It's been very helpful
for running apps that depend on a persistent FS, like Wordpress. Perhaps
this use case of mounting a remote SSHFS could be supported in some other
way?

Dan








*From: *Onsi Fakhouri
*Reply-To: *"Discussions about Cloud Foundry projects and the system
overall."
*Date: *Saturday 11 July 2015 01:10
*To: *cf-dev
*Subject: *[cf-dev] Removing FUSE support from CF



Hey CF-Dev,



The Garden team has been hard at work substantially improving
Garden-Linux's security features. Garden-Linux now employs user namespaces
and drops capabilities when creating unprivileged containers - we're
excited to bring both of these features to the platform!



Diego currently runs applications in *privileged* containers. These
lack the security features outlined above and we are planning on switching
to launch all CF applications in *unprivileged* containers.



Unfortunately, it has proved difficult to support
mounting FUSE filesystems from within unprivileged containers. We believe
the security benefits outweigh the features that FUSE give us and* are
planning on removing support for FUSE in favor of better securing our
containers.* If/when FUSE support in unprivileged containers becomes
possible we may add it back to the platform.



Thoughts? Concerns?



Thanks!



Onsi

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
-- Matt


Re: Notifications on ORG, SPACE and USER modifications

Matt Cowger
 

For me the primary use case to developing management tools on top of the CC
API, and being able to intelligently cache information about the state of
those resources. Without etags or serial numbers or similar, that means
every refresh needs to query a TON of data from CC.

On Tue, Jul 28, 2015 at 10:26 PM, Dieu Cao <dcao(a)pivotal.io> wrote:

Hi all interested in notifications on modification of resources,

It would be helpful for me in framing the "what" and the "why" of this
feature if you could also describe your specific use cases and pain points
on why you would want notifications on modifications and also which
resources you particularly care about.
Is it for real time updates on a dashboard? For consumption for billing
purposes? For triggering provisioning/deprovisioning of resources?

-Dieu

On Tue, Jul 28, 2015 at 11:05 AM, Jean-Sebastien Delfino <
jsdelfino(a)gmail.com> wrote:

I’m going to need something like this too for the CF Abacus service
metering project, as I’d like to track the lifecycle of orgs, users, etc to
match their history with the usage data reported for them.


Here’s a straw man description of what I had in mind:


- For Abacus, I’d need a Lossless API. Usage metering eventually
translates to billing and money, you don’t want to lose that :)


- An extension or variant of the current CF /v2/events API supporting
users, orgs, app usage etc, as even with a notification API I’ll still need
to do GETs sometimes.


- 304 responses with etags on these GETs (as suggested earlier in the
thread [1]) would be good.


- A Webhook style notification API where I could register interest in a
selection of events with a callback URL, and get these events POSTed back
to me at that URL, similar to what Github and many others do with Webhooks.


- On top of Webhooks, it’d be nice to have a form of streaming (either
down to the client like the Firehose does or in the other direction up to
the Webhook callback URL), but I'm not sure if we’ll need that in the
project right away.


- We’d obviously need some form of security, maybe use my user token to
register for events on entities that I have access to?


- I’m also curious about the group’s thoughts on queueing and
back-pressure when events get generated faster that they can be consumed
for example. There was a mention of some message queuing earlier [2]. That
would make sense to me (although IMO it’d be good if the underlying MQ
didn’t shine through the API). What did you have in mind for this?


I guess there are quite a few things to figure out here! I’ll be happy to
collaborate with the community on these discussions.


Thoughts?


[1]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p842.html

[2]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tp827p834.html


- Jean-Sebastien

On Fri, Jul 24, 2015 at 9:59 PM, Matt Cowger <matt(a)cowger.us> wrote:

I think ETags is reasonable thought as well.

On Thu, Jul 23, 2015 at 4:39 PM, Benjamin Black <bblack(a)pivotal.io>
wrote:

ETags and a 304 response are specifically intended for that purpose.
I'd recommend that over relying on Last-Modified.


b

On Thu, Jul 23, 2015 at 12:34 AM, Koper, Dies <
diesk(a)fast.au.fujitsu.com> wrote:

Or setting the Last-Modified HTTP response header accordingly, and
allow clients to use HTTP caching mechanisms (Last-Modified, etc.) to get
quick empty responses with the current APIs if no changes have been made?
(Or maybe this is already working so – haven’t checked).



Regards,

Dies Koper



*From:* cf-dev-bounces(a)lists.cloudfoundry.org [mailto:
cf-dev-bounces(a)lists.cloudfoundry.org] *On Behalf Of *Matt Cowger
*Sent:* Thursday, July 23, 2015 4:45 PM
*To:* Discussions about Cloud Foundry projects and the system overall.
*Subject:* Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications



I've wanted something similar as well.



On a related note, having a CC API 'serial' number (for each object in
CC - apps, spaces, etc) that increments on every change relevant to that
object would be of value for detecting if something has changed.



On Thu, Jul 23, 2015 at 3:27 PM, Dieu Cao <dcao(a)pivotal.io> wrote:

There are a few different approaches to this and different concerns
that are possible.

The requests I've seen have been around wanting to be able to
subscribe to and filter the various events that cc currently generates so
that other behavior could be triggered.

We currently have events, app usage events, and service usage events.

Is it acceptable for the notifications to be lossy? Depends on the
use case but If so, then the firehose may be an acceptable approach.



The CAPI team is currently focusing on other work in the near term,
such as the v3 API and private brokers, but would be happy to collaborate
on a proposal.





On Wed, Jul 22, 2015 at 2:05 PM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:

My take:



CC should have callbacks on for each model create, update and delete
methods. Those callbacks will send a message to an MQ, which you can
subscribe to consume those messages.

This can be expanded to pretty much every event we need to track.

What do you think?



JP



2015-07-22 17:30 GMT-03:00 Matthias X Hub <matthias.hub(a)de.ibm.com>:

Hi,

we (=IBM) are also having the need and are currently investigating how
to solve this. We plan to work on a proposal to discuss this further with
the cf community. I'll keep you updated on that.

Regards,
Matthias



From: Mike Youngstrom <youngm(a)gmail.com>
To: "Discussions about Cloud Foundry projects and the system
overall." <cf-dev(a)lists.cloudfoundry.org>
Date: 22.07.2015 20:57
Subject: Re: [cf-dev] Notifications on ORG, SPACE and USER
modifications
Sent by: cf-dev-bounces(a)lists.cloudfoundry.org
------------------------------




We have the same need. Today we are polling the CC.

It would be nice for us also if we could get CC event notifications
via something like the firehose.

Mike

On Wed, Jul 22, 2015 at 10:23 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:
I mean, I know you can list those events thru the API, but I want
something that will react on an event instead of having to be constantly
polling for them.

2015-07-22 13:18 GMT-03:00 Juan Pablo Genovese <
juanpgenovese(a)gmail.com>:
Sree,

thanks! Any pointers on how can I hook up to these audit events?

Thank you!

2015-07-22 13:12 GMT-03:00 Sree Tummidi <stummidi(a)pivotal.io>:
I believe there are audit events generated for all these actions which
can be captured and forwarded to an SIEM solution like splunk

Thanks,
Sree

Sent from my iPhone

On Jul 22, 2015, at 8:54 AM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:

Guys,

I need to somehow hook up into the Cloud Controller (CC) to capture
ORG, SPACE and USER deletion, insertion and update.

So far, I considered some approaches, such as forking the CC (the
least favorite) and modifying the code with some hooks, tapping into Nginx
to capture the requests, and using triggers in the database to capture each
event and send the necessary info to a service.

What do you think?
Any other idea you might have?

Thanks!

--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------

http://www.jpgenovese.com
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com



--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev





--

Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------

http://www.jpgenovese.com


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev





--

-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
-- Matt

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Jean-Sebastien

Sent from my DynaTAC 8000x

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
-- Matt


Droplets and Stacks

Colin Humphreys <colin@...>
 

Hi All,

I wrote a couple of articles about droplets and stacks.

http://www.cloudcredo.com/a-droplet-of-value/

http://www.cloudcredo.com/stacks-of-problems/

The droplet post is fairly self-explanatory, and enabling the choice of
shipping droplets or source is well on the way in Cloud Foundry development.

I feel our story around stacks is far less complete. It seems to be an
overloaded concept inherited from Heroku and the contract with the stack
seems to cause issues for both app and buildpack developers.

I'd like to open the discussion on what the future should be for stacks, or
if you think they're perfect as they are.

Cheers,
Colin

CloudCredo Cheerleader

8401 - 8420 of 9398