Date   

Re: Immutability of applications

john mcteague <john.mcteague@...>
 

Marco, So I understand that configuration for an app is mutli-faceted and
not the complete view of what is happening. My goal is to understand what
is running at any moment in time from the information the API provides me.

Buildpack's can be heavily influenced by env var changes. Developers can
affect the version of java or the JVM args via env vars, but it is not
possible via the API to be sure from the variables presented whether they
are those that actually took effect at startup/restage. This causes an
audit concern.

Where an app adjusts its settings based on Redis being bound for example,
that is auditable by the fact we wrote the app and know what is due to
happen upon binding. But because binding can occur on a running app and
that is not guaranteed to be effective immediately, from the API output,
the developer may assume the application is running with that Redis
specific configuration. But that is only true if the app was restarted.

CF cant control every aspect of the apps configuration, but where it is the
method of configuration (bindings, env vars), I feel its important that
appropriate distinctions between actual and desired state are made.

On Tue, Nov 10, 2015 at 5:08 PM, Marco Nicosia <mnicosia(a)pivotal.io> wrote:

John,

I'd like to understand more about what you are hoping to accomplish.

Are you trying to detect gaps where a new env var has been pushed, but the
app has not been restarted? Is this for monitoring/compliance or debugging
purposes?

It seems like Apps these days get configuration from a pretty wide variety
of sources, env variables being only one. Apps also ship with defaults,
UNIX defaults (max filedes, semaphores, etc), and apps sometimes adjust
their settings based on values in SQL or Redis*, etc.

Each app cares about these settings differently, some are sensitive to RAM
available, but not number of open files, etc.

I only ask because it seems like asking for "effective environment" vs
"actual environment" is only a subset of configurations.

The infrastructure can only express what has been provided to the app, but
for any number of reasons, may not represent the current actual running
state of the app.

I understand that it means more work for the app dev, but I often
implement a quick "/config" endpoint in my apps. This allows me to verify
what configuration the App is actually using at a given moment.

It's a more comprehensive way of validating that all "intended
configuration," is in fact actual configuration.

--
Marco Nicosia
Product Manager
Pivotal Cloud Foundry

* A quick example: the NOC can use an admin endpoint to toggle on/off an
experimental feature, which is detected via Redis, etc.

On Monday, November 9, 2015, Dieu Cao <dcao(a)pivotal.io> wrote:

Hi John,

I had been thinking about this a bit more, and I think it would be
reasonable to consider adding an end point that is similar to the /stats
end point, that queries Diego, to get actual state on environment variables
and as part of that VCAP_SERVICES for the running app instances.
Thoughts on that approach? Would that help address the problem?

-Dieu

On Wed, Nov 4, 2015 at 9:03 AM, john mcteague <john.mcteague(a)gmail.com>
wrote:

I had this conversation with a few different people during the berlin
summit and promised one of them I would repeat it on the mailing list today
to get further feedback.

Today, once we push an application, the droplet is immutable. It doesnt
change until you push the application again or restage. I believe the
entire container could change without a new push if you upgrade the rootfs
and restart all the apps (which the CF operator would do).

However, the environment vars and service bindings can be changed on an
application but they would not take affect until the next restart.The CF
API would report these changes as active when you run *cf env *or *cf
services. *There is no distinction between desired state and current
state when using the API.

To me this is a significant gap as we cannot necessarily get a true view
of the world (i call cf set-env but dont restart the app, how do I know
from the API what value of that env var my app is using).

How are people addressing this in their own environments and is it
something that the core API team should be considering (I ask the latter
publicly even though I asked Dieu during the summit :) ).

John
--
--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io
c: 650-796-2948



Re: Notifications for service brokers

Jean-Sebastien Delfino
 

Hi all,

I'd like to start looking into how to leverage these notifications in the
Abacus project.

Has there been any progress on this?

Thanks

- Jean-Sebastien

On Fri, Sep 4, 2015 at 7:58 AM, Dieu Cao <dcao(a)pivotal.io> wrote:

To follow up on this, I've been working with Simon Moser on an initial
proposal for this and he is now taking lead on it. Simon just completed a
PM dojo at the end of August.

Dieu


On Tuesday, August 18, 2015, Dieu Cao <dcao(a)pivotal.io> wrote:

I planned to put together a proposal for this a couple of weeks ago as a
strawman to describe use cases, but just have not had the time.
I still hope to tackle this in the next week or so and will post to this
list.

For reference, see this thread [1] where this was previously discussed.

-Dieu
CF CAPI PM

[1]
http://cf-dev.70369.x6.nabble.com/cf-dev-Notifications-on-ORG-SPACE-and-USER-modifications-tt827.html#none

On Tue, Aug 18, 2015 at 5:47 PM, Vineet Banga <vineetbanga1(a)gmail.com>
wrote:

Thanks Juan, I will try to setup a poller for this to achieve similar
functionality. Do you know if there is already proposal for the better
notifications - if yes, could you point me to it? I Would like to see if
it would meet our needs at some point in the future.

On Fri, Aug 14, 2015 at 4:26 PM, Juan Pablo Genovese <
juanpgenovese(a)gmail.com> wrote:

Vineet,

there is some proposals to add better notifications to CF in general
and the CC in particular, but for now you can poll the CC API to get those
events. See http://apidocs.cloudfoundry.org/214/

Thanks!

2015-08-14 18:31 GMT-03:00 Vineet Banga <vineetbanga1(a)gmail.com>:

Is there any notification pub/sub mechanism in cloud foundry when
services are created/updated/deleted. We are exposing few services in CF
using service brokers and we would like some common actions to occur when
our services are created/delete/updated.


--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com


Re: [abacus] CouchDB

Jean-Sebastien Delfino
 

Hi Hristo,

I think it should be fairly easy to create another Abacus dbclient module
targeting another DB. AFAIK we're only doing simple get/put/remove, bulk
gets/puts and a few range queries over the doc keys, and do not depend on
more sophisticated CouchDB features like design docs, map reduce views etc,
so porting that to another key/json-value DB (even a SQL DB like Postgres
or MySQL) should be possible.

While exploring another DB, you may also want to think about the level of
DB partitioning you'd like (e.g. one DB forever, monthly DBs, some sharding
etc) as with usage data we're getting into much bigger volumes than what
people usually store in CCDB for example, but that partitioning logic is
also easily customizable.

I'll also be happy to adjust a few things around the code base where we
call that dbclient module if needed, if you guys run into limitations or
issues when porting to another DB.

- Jean-Sebastien

On Tue, Nov 10, 2015 at 8:00 AM, Hristo Iliev <hsiliev(a)gmail.com> wrote:

Hi,

I'm wondering how tightly Abacus is bound to CouchDB? Can we create
another dbclient that stores the data in DB that allows raw json documents?
How easy/hard this would be and where one should start?

The backgound is that using another DB will help integrators maintain only
a handful of DBs. Having lots of backing services (DBs) usually requires a
lot of resources (VMs, memory, support) and is quite expensive. Reducing
the number of DBs also makes it easy to move a solution especially in the
case of private clouds :)

Regards,
Hristo Iliev


Re: [abacus] Eureka vs gorouter

Jean-Sebastien Delfino
 

Hey Hristo,

Eureka helps us as an optional registry for monitoring the performance of
each of our apps with a Hystrix [1] dashboard. We're not using it for
routing the flow of usage events across our apps or load balancing or
anything like the CF Go router does.

We found Hystrix pretty useful to better understand the performance
characteristics of the Abacus apps under load (latency, timeouts, error
rates etc). There are several ways to set up Hystrix to monitor Cloud apps,
but Eureka comes handy when you don't know their IP addresses ahead of
time. The usual setup is then to use Eureka + Turbine + Hystrix (as
described in [2]). You get your apps to register with Eureka, set up
Turbine to get their IPs from Eureka, and serve an aggregated performance
data stream to your Hystrix dashboard for all your apps.

That's what we've been experimenting with, and I believe that Assk
(@sasrin) has started to document the beginning of that monitoring setup as
well in doc/monitor.md [3]. BTW this is all optional, as we only register
the apps if you've set up a Eureka env variable.

HTH

P.S. On a different topic, speaking of the Go router... it actually gets in
the way sometimes. For example, say I want to monitor, poke at, or test a
specific instance of app X running on CF, or each individual instance of
app X... AFAIK I can't do that in front of the Go router as it's not
letting me open HTTP connections to a specific instance of my app (unless I
play around with sticky sessions but that's really just a hack). So to get
around that I often need to struggle to deploy our monitors, debuggers,
HTTP perf tools etc inside CF, behind that Go router... I will probably
raise that issue to the CF community in a different non-Abacus thread... as
I've been finding that pretty inconvenient.

[1] https://github.com/Netflix/Hystrix/tree/master/hystrix-dashboard
[2] http://techblog.netflix.com/2012/12/hystrix-dashboard-and-turbine.html
[3]
https://github.com/cloudfoundry-incubator/cf-abacus/blob/master/doc/monitor.md

- Jean-Sebastien

On Tue, Nov 10, 2015 at 7:54 AM, Hristo Iliev <hsiliev(a)gmail.com> wrote:

Hi,

Recently abacus introduced Eureka client and stub. Looking at the high
level description of what Eureka does it seems it has almost the same
responsibilities as gorouter.

What is the reason to have another loadbalancing layer? Can you elaborate
on why it was introduced, what problems it solves and why can't gorouter be
used for the same?

Regards,
Hristo Iliev


Re: Immutability of applications

Marco Nicosia
 

John,

I'd like to understand more about what you are hoping to accomplish.

Are you trying to detect gaps where a new env var has been pushed, but the
app has not been restarted? Is this for monitoring/compliance or debugging
purposes?

It seems like Apps these days get configuration from a pretty wide variety
of sources, env variables being only one. Apps also ship with defaults,
UNIX defaults (max filedes, semaphores, etc), and apps sometimes adjust
their settings based on values in SQL or Redis*, etc.

Each app cares about these settings differently, some are sensitive to RAM
available, but not number of open files, etc.

I only ask because it seems like asking for "effective environment" vs
"actual environment" is only a subset of configurations.

The infrastructure can only express what has been provided to the app, but
for any number of reasons, may not represent the current actual running
state of the app.

I understand that it means more work for the app dev, but I often implement
a quick "/config" endpoint in my apps. This allows me to verify what
configuration the App is actually using at a given moment.

It's a more comprehensive way of validating that all "intended
configuration," is in fact actual configuration.

--
Marco Nicosia
Product Manager
Pivotal Cloud Foundry

* A quick example: the NOC can use an admin endpoint to toggle on/off an
experimental feature, which is detected via Redis, etc.

On Monday, November 9, 2015, Dieu Cao <dcao(a)pivotal.io> wrote:

Hi John,

I had been thinking about this a bit more, and I think it would be
reasonable to consider adding an end point that is similar to the /stats
end point, that queries Diego, to get actual state on environment variables
and as part of that VCAP_SERVICES for the running app instances.
Thoughts on that approach? Would that help address the problem?

-Dieu

On Wed, Nov 4, 2015 at 9:03 AM, john mcteague <john.mcteague(a)gmail.com
<javascript:_e(%7B%7D,'cvml','john.mcteague(a)gmail.com');>> wrote:

I had this conversation with a few different people during the berlin
summit and promised one of them I would repeat it on the mailing list today
to get further feedback.

Today, once we push an application, the droplet is immutable. It doesnt
change until you push the application again or restage. I believe the
entire container could change without a new push if you upgrade the rootfs
and restart all the apps (which the CF operator would do).

However, the environment vars and service bindings can be changed on an
application but they would not take affect until the next restart.The CF
API would report these changes as active when you run *cf env *or *cf
services. *There is no distinction between desired state and current
state when using the API.

To me this is a significant gap as we cannot necessarily get a true view
of the world (i call cf set-env but dont restart the app, how do I know
from the API what value of that env var my app is using).

How are people addressing this in their own environments and is it
something that the core API team should be considering (I ask the latter
publicly even though I asked Dieu during the summit :) ).

John
--
--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io
c: 650-796-2948


[abacus] CouchDB

Hristo Iliev
 

Hi,

I'm wondering how tightly Abacus is bound to CouchDB? Can we create another dbclient that stores the data in DB that allows raw json documents? How easy/hard this would be and where one should start?

The backgound is that using another DB will help integrators maintain only a handful of DBs. Having lots of backing services (DBs) usually requires a lot of resources (VMs, memory, support) and is quite expensive. Reducing the number of DBs also makes it easy to move a solution especially in the case of private clouds :)

Regards,
Hristo Iliev


[abacus] Eureka vs gorouter

Hristo Iliev
 

Hi,

Recently abacus introduced Eureka client and stub. Looking at the high level description of what Eureka does it seems it has almost the same responsibilities as gorouter.

What is the reason to have another loadbalancing layer? Can you elaborate on why it was introduced, what problems it solves and why can't gorouter be used for the same?

Regards,
Hristo Iliev


Re: [buildpacks] CF-built binaries: release timeline and beta testing

Mike Dalessio
 

Hi Shawn,

Replies inline.



On Mon, Nov 9, 2015 at 7:12 PM, Shawn Nielsen <sknielse(a)gmail.com> wrote:

Mike,

Thanks for the quick reply.

My question is more around using an 'online' buildpack vs. 'offline'.
When I say 'online', I'm referring to the binaries being pulled from the s3
repo at staging time.
Right, but in your question you are specifically stating that the "offline"
(or "uncached") buildpacks has different URLs than the "online" (or
"cached"), and I want to make my point clear that this is not the case.

The URLs are identical between online and offline behavior for any specific
version of the buildpack. I believe you're comparing the behaviors of two
different versions of the buildpack.



If we use the offline buildpack binaries, we are restricted to node
versions explicitly defined in the manifest files.
This is intentional. It's the Buildpack policy to, *by default*, ship only
the two most recent releases on a major/minor branch, to help ensure that
developers using CF are updating to the most secure versions of these
binaries. This was announced earlier in the year in an extensive email
thread
<https://groups.google.com/a/cloudfoundry.org/d/msg/vcap-dev/1HmGK4wU3Rc/lk186OOtdbMJ>
about reducing the sizes of buildpacks, while improving the security level
of the default platform.

You're free to change this policy for your CF deployment. Details are below.


This creates some issues when projects want to do validation testing
between one version and the next or need to roll back for whatever reason.
I understand this issue, and I empathize. In my opinion, the current *default
policy* is the is the least-pessimal (meaning, the least bad) solution
available to us. Including more binaries increases the vulnerability
surface area of the platform as well as slowing down deployment and staging
times. Including fewer binaries makes upgrades nearly impossible.

Again, you're free to change this policy for your CF deployment. Details
are below.

The community raised no objections to this policy, which leads me to
believe that this opinion is at least reasonably defensible.


It also requires us to manually update the buildpack every time there is a
new release. We've historically used the online buildpacks to prevent
these types of issues, but we'd like to better understand Pivotal's long
term strategy here.
I am an open-source PM of an open-source project, and we've been completely
transparent about the reasoning and logic behind this decision. We asked
the community for comments over an extended period of time, and received no
objections to implementing this policy.

Further, *we've open-sourced all the tooling that CF operators need to
implement a different buildpack policy*. If you feel the default policy
doesn't match the requirements of your organization, you are fully enabled
to create a custom buildpack with a custom set of binaries that match your
organization's requirements.

If you'd like help implementing your own buildpack releases, the
open-source Buildpacks team would be more than happy to help.

If you'd like to know more, you might start here:

https://github.com/cloudfoundry/buildpack-packager
and if you want to generate custom binaries, we can help with that, too:

https://github.com/cloudfoundry/binary-builder
Let me know how this open-source team can help you meet your company's
requirements.



Going forward, what is Pivotal's strategy for online binaries that would
be used at staging time?
-Shawn

On Mon, Nov 9, 2015 at 4:03 PM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Hi Shawn,

Thanks for asking.

The actual URL used is going to depend on what version of the buildpack
you're using. Currently master (v1.5.1) is only referncing CF-generated
binaries:

https://github.com/cloudfoundry/nodejs-buildpack/blob/master/manifest.yml

We removed references to s3pository.heroku.com in February, for v1.2.0
of the buildpack.

Let me know if you need more context around this. Thanks!

-mike


On Mon, Nov 9, 2015 at 4:40 PM, Shawn Nielsen <sknielse(a)gmail.com> wrote:

James,

It looks like the online NodeJS buildpack is currently pointing to
Heroku's repo:
http://s3pository.heroku.com/node/...

Whereas the offline NodeJS buildpack manifest files seems to be pointing
to pivotal's repo:
https://pivotal-buildpacks.s3.amazonaws.com/concourse-binaries/node/...

Is there a reason for this discrepency?

I would think the online NodeJS buildpack should be pointing to the same
pivotal repo as the offline:
https://pivotal-buildpacks.s3.amazonaws.com/concourse-binaries/node/...


Let me know if you have any input here. Thanks,

-Shawn


On Fri, Jul 17, 2015 at 11:46 PM, James Bayer <jbayer(a)pivotal.io> wrote:

mike d is the best to answer, but i'll take a crack at it

On Fri, Jul 17, 2015 at 3:24 PM, Shawn Nielsen <sknielse(a)gmail.com>
wrote:

Two questions on these cf-built binaries:

1. From my understanding, the purpose of this is to compile the
binaries so they are optimized for the cf specific stacks (e.g.
cflinuxfs2 ) as opposed to something that was generically compiled
(e.g. to 64 bit trusty). Can you confirm this or expound upon this if
there are other reasons?
for some buildpacks, we have been relying on heroku binaries which is
an external dependency. we want the cf team to be in complete control of
how and when the binaries are built. this ensures cf can be in control of
our own destiny when for patching security issues or bugs. additionally it
means cf can take responsibility for how the binaries are compiled to
increase trust of the binary contents.



2. Are these binaries available in offline mode only or is there also
intent that they will be hosted, allowing us to consume them in an online
mode?
most/all cf buildpacks are available in online and offline mode as i
understand it. see:
https://github.com/cloudfoundry-incubator/buildpack-packager/blob/master/doc/disconnected_environments.md



Thanks,

-Shawn

On Mon, Jul 13, 2015 at 2:08 PM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:



On Mon, Jul 13, 2015 at 1:08 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Wed, Jul 8, 2015 at 1:55 PM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Hi all,

In the June CAB call, it was mentioned that the Buildpacks team was
working on generating CF-specific binaries to be packaged in the
buildpacks. I'm pleased to announce that the team is ready to start
shipping these binaries in the golang, nodejs, php, python, and ruby
buildpacks for the cflinuxfs2 stack.

*__We're planning to cut releases of these buildpacks with the CF*
*binaries on Monday, 20 July.__*

In the meantime, I'd like to ask the community to beta these
buildpacks and give us feedback.

We're successfully running [BRATs][1] (the buildpack runtime
acceptance tests) on these binaries, so we don't expect any issues; but
we'd love to hear if anyone experiences any issues deploying their apps.

Obviously, until we cut official releases, you should use judgement
when deciding where to use these beta buildpacks.

[1]: https://github.com/cloudfoundry/brats


*## Timeline, Versioning and Proposed Changes*

Unless we hear of any blocking issues, we'll cut official releases
of the go, node, php, python and ruby buildpacks on July 20th.

When we cut these releases, we'll be bumping the major version
number, and removing the `manifest-including-unsupported.yml` file from the
repository HEAD. I'd love to hear anyone's opinion on these changes as well.


*## How to deploy with the "beta" binary buildpacks*

Until we cut official releases, we are maintaining a git branch in
each buildpack repository, named `cf-binary-beta`, in which the manifest
points at our CF-specific binaries.

If your CF deployment has access to the public internet, you can
push your app and specify the github repo and branch with the `-b` option.

*__Go:__*

`cf push <appname> -b
https://github.com/cloudfoundry/go-buildpack#cf-binary-beta`
<https://github.com/cloudfoundry/go-buildpack#cf-binary-beta>

*__Node:__*

`cf push <appname> -b
https://github.com/cloudfoundry/nodejs-buildpack#cf-binary-beta`
<https://github.com/cloudfoundry/nodejs-buildpack#cf-binary-beta>

*__PHP:__*

`cf push <appname> -b
https://github.com/cloudfoundry/php-buildpack#cf-binary-beta`
<https://github.com/cloudfoundry/php-buildpack#cf-binary-beta>
Looks good mostly. One minor issue. It seems like the snmp
extension is not loading correctly. Looks to be missing a required shared
library.

2015-07-13T12:29:53.09-0400 [App/0] OUT 16:29:53 php-fpm |
[13-Jul-2015 16:29:53] NOTICE: PHP message: PHP Warning: PHP Startup:
Unable to load dynamic library
'/home/vcap/app/php/lib/php/extensions/no-debug-non-zts-20100525/snmp.so' -
libnetsnmp.so.30: cannot open shared object file: No such file or directory
in Unknown on line 0

That's with PHP 5.4.42, 5.5.26 and 5.6.10.
Good catch, I've created a story for this:
https://www.pivotaltracker.com/story/show/98980384





One other thing, which is not really an issue, but perhaps an
optimization. I noticed for the PHP extensions the bundle has both ".a"
and ".so" files for many of the extensions. The ".a" static libraries
should not be necessary, just the ".so" shared libraries. Seems like
removing them could save 8 to 9M. You could save another 25M by deleting
the bin/php-cgi file. You really only need bin/php for cmd line stuff and
sbin/php-fpm for web apps. Can't think of any reason you'd need / want to
do cgi. It's not a ton of space, 35M X 3 (each current version) + 35M X 3
(each of previous versions) - whatever compression will save. Just thought
I'd throw it out there though.

Another good catch, I've created a story to look into it:
https://www.pivotaltracker.com/story/show/98980692




Dan




*__Python:__*

`cf push <appname> -b
https://github.com/cloudfoundry/python-buildpack#cf-binary-beta`
<https://github.com/cloudfoundry/python-buildpack#cf-binary-beta>

*__Ruby:__*

`cf push <appname> -b
https://github.com/cloudfoundry/ruby-buildpack#cf-binary-beta`
<https://github.com/cloudfoundry/ruby-buildpack#cf-binary-beta>


If your CF deployment doesn't have access to the public internet
and you'd like to try these buildpacks, please reach out directly and we'll
figure out the best way to accommodate.


*## Tooling Details*

If you'd like to take a look at how we're currently building these
binaries, our tooling is open-sourced at:

https://github.com/cloudfoundry/binary-builder
Note that `binary-builder` uses the rootfs docker image to compile
each binary, which means that this tool can easily be extended to
essentially cross-compile **any** binary for a CF rootfs. We'd
love to hear your feedback on this, as well.


Thanks,
-mike



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Utilities PMC - 2015-11-09 Notes

Mike Dalessio
 

Hello CF Community!

Here is an update from the Utilities PMC, as of 2015-11-09. The full notes
are available at:

https://github.com/cloudfoundry/pmc-notes/blob/master/Utilities/2015-11-09-utilities.md

but I've reproduced them below for your convenience.

Please also join me in congratulating *Dies Köper*, of Fujitsu, who has
taken over as the PM of the CLI team after a "PM Dojo" that ended on
October 29. (*hold for applause*) Many of you know Dies from his presence
in the community and his previous contributions to Cloud Foundry, and so he
likely does not need a formal introduction.

Greg Oehmen, who previously filled the PM position for CLI, has moved into
a new role within Pivotal. I'd like to personally thank both Dies and Greg
for their amazing efforts on Cloud Foundry, and I'm looking forward Dies
helping to make a great product even better.

Cheers,
-mike

---

CLI

Dies Köper, of Fujitsu, has taken over the PM responsibilities for the CLI
team after completing a "PM Dojo" on October 29.

Dies has embarked on cleaning up the icebox (evaluating around 300 old
stories) as well as going through open Github Issues and PRs.

Congratulations, and thank you, Dies!
<https://github.com/cloudfoundry/pmc-notes/blob/master/Utilities/2015-11-09-utilities.md#releases>
Releases

CLI v6.13.0 was released, which pulls Diego support into the main tool
(i.e., the Diego plugin is no longer needed).
<https://github.com/cloudfoundry/pmc-notes/blob/master/Utilities/2015-11-09-utilities.md#ongoing--upcoming-work>Ongoing
/ Upcoming work

- Started implementing stories around Organization and Space RBAC
<https://www.pivotaltracker.com/epic/show/1991856>.
- Started fleshing out stories upcoming around CC v3 API
<https://www.pivotaltracker.com/epic/show/2088038>.
- Implemented a number of First Impressions stories:
- Simplification of the Download page
<https://github.com/cloudfoundry/cli#downloads> (making 64 bit
installers and binaries more prominent than the edge and 32 bit releases)
- Examples on how to download them with curl
- Improved filenames (added release version in filename, etc.)

<https://github.com/cloudfoundry/pmc-notes/blob/master/Utilities/2015-11-09-utilities.md#risks>
Risks

- Team resource levels unstable, with one engineer added and two rolled
off. This brings up the risk of loss of knowledge and context.
- Many days this fortnight we had only 1 engineer working (others were
sick, on vacation, had jury duty calls). Low staffing to continue with
anchor visiting upcoming CF Summits.

<https://github.com/cloudfoundry/pmc-notes/blob/master/Utilities/2015-11-09-utilities.md#java-utilities>Java
Utilities

The CF Java client work has slowed down as the team ramps up on Concourse,
but is continuing to proceed. Interested parties are invited to follow
progress in the Java Client backlog
<https://www.pivotaltracker.com/n/projects/816799>.


Buildpacks PMC - 2015-11-09 Notes

Mike Dalessio
 

Hello CF community!

Here is an update from the Buildpacks PMC, as of 2015-11-09. The full notes
are available at:

https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md

but I've reproduced them below for your convenience.

Note that in addition to the releases below, we'll also be cutting a
cflinuxfs2 release today to include a recent CVE update.

Cheers,
-mike

----

Stacks

We've been continuing to make updates to the cflinuxfs2 rootfs at least
weekly, with additional turnaround of CVE updates within 48 hours.
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#releases>
Releases

Released cflinuxfs2 versions 1.10.0
<https://github.com/cloudfoundry/stacks/releases/tag/1.10.0>, 1.11.0
<https://github.com/cloudfoundry/stacks/releases/tag/1.11.0>, 1.12.0
<https://github.com/cloudfoundry/stacks/releases/tag/1.12.0>, 1.13.0
<https://github.com/cloudfoundry/stacks/releases/tag/1.13.0>, 1.14.0
<https://github.com/cloudfoundry/stacks/releases/tag/1.14.0>, and 1.15.0
<https://github.com/cloudfoundry/stacks/releases/tag/1.15.0>, addressing:

USN-2788-1 <http://www.ubuntu.com/usn/usn-2788-1> "unzip vulnerabilities",
which is related to:

- CVE-2015-7696 "Heap buffer overflow when extracting password-protected
archive"
- CVE-2015-7697 "Infinite loop when extracting password-protected
archive"

USN-2787-1 <http://www.ubuntu.com/usn/usn-2787-1>, "audiofile
vulnerability", which is related to:

- CVE-2015-7747 "made to crash or run programs as your login if it
opened a specially crafted file"

USN-2767-1 <http://www.ubuntu.com/usn/usn-2767-1>, "GDK-PixBuf
vulnerabilities", which is related to:

- CVE-2015-7673
<http://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-7673.html>
"Heap
overflow and DoS with a tga file in gdk-pixbuf < 2.32.1"
- CVE-2015-7674
<http://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-7674.html>
"Heap
overflow with a gif file in gdk-pixbuf < 2.32.1"

<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#buildpacks>
Buildpacks
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#general>
General

Continuing on the track of work to provide a "chain of custody"
<https://www.pivotaltracker.com/epic/show/2077742> for everything shipped
in the buildpacks, each buildpack released is now published with a SHA256
checksum in the release notes.
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#go-buildpack>
go-buildpack
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#releases-1>
Releases

Released v1.6.3
<https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.6.3> adding
support for Go 1.4.1 for upgrade paths, adding a Godep binary to the
manifest, and supporting the linker's -X option format for go1.5.
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#proposals>
Proposals

In the last set of PMC notes, it was proposed to drop golang v1.2.x and
v1.3.x <https://github.com/cloudfoundry/go-buildpack/issues/22> and no
objections were raised. We'll be scheduling work on this over the next few
weeks.
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#notes>
Notes

We're also in the process of sending pull requests back upstream to godep and
to the go-buildpack to allow "wildcarding" of golang versions, meaning that
"1.4.*" would use the most recent matching golang version. This would make
more palatable the "skinny buildpack" policy of only providing the two
most-recent supported versions on a major/minor branch. See GH#22
<https://github.com/cloudfoundry/go-buildpack/issues/22> for details.
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#java-buildpack>
java-buildpack
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#releases-2>
Releases

Released v3.3.1
<https://github.com/cloudfoundry/java-buildpack/releases/tag/v3.3.1>. This
release contains a new debug framework and ensures that the dependencies
contained in theoffline buildpack are up to date.
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#nodejs-buildpack>
nodejs-buildpack
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#releases-3>
Releases

Released v1.5.1
<https://github.com/cloudfoundry/nodejs-buildpack/releases/tag/v1.5.1>,
which adds support for Node 4.2.x LTS, emits buildpack details from
bin/detect, and merges many commits from upstream.
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#proposals-1>
Proposals

In the last PMC notes, it was proposed to add node.js 4.x support
<https://github.com/cloudfoundry/nodejs-buildpack/issues/32> via static
linking of dependencies. This work was completed and shipped in
nodejs-buildpack v1.5.1 as noted above.
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#php-buildpack>
php-buildpack
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#releases-4>
Releases

Released v4.2.1
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v4.2.1>, v4.2.0
<https://github.com/cloudfoundry/php-buildpack/releases/tag/v4.2.0>, and
v4.1.5 <https://github.com/cloudfoundry/php-buildpack/releases/tag/v4.1.5>,
which collectively update nginx to v1.9.6, update httpd to 2.4.17, add
support for PHP 5.6.16 and 5.5.30, and drop support for PHP 5.4.x (which
has been EOLed).
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#proposals-2>
Proposals

In the last PMC notes, it was proposed to remove nginx 1.6
<https://github.com/cloudfoundry/php-buildpack/issues/109> from the
buildpack, but continuing to support nginx 1.8 and 1.9, and no objections
were raised. We'll schedule work on this change.
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#ruby-buildpack>
ruby-buildpack
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#releases-5>
Releases

Released v1.6.8
<https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.6.8>, which
updates JRuby to 9.0.3.0, and updates OpenJDK to 1.8.0_65.
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#staticfile-buildpack>
staticfile-buildpack
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#releases-6>
Releases

Released v1.2.2
<https://github.com/cloudfoundry/staticfile-buildpack/releases/tag/v1.2.2>,
which improves error messages and log messages, and emits buildpack details
from bin/detect.
<https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-11-09-buildpacks.md#notes-1>
Notes

Some performance analysis was done of the staticfile-buildpack after
cf-release v221's increase of ip_conntrack_max, and the results may be
interesting to readers. Details are in this tracker story
<https://www.pivotaltracker.com/story/show/105014548>, but the TL;DR is
that we saturated the testing network before we hit nginx bottlenecks. Peak
performance for 1KB payloads was in excess of 1400 requests/second


Re: CF CAB call for November is this Wednesday Nov. 11th, 2015

Michael Maximilien
 

Of course I meant November in note. Talk soon. Call info in the agenda.




Best,




Max




Sent from Mailbox

On Tue, Nov 10, 2015 at 2:12 AM, Michael Maximilien <maxim(a)us.ibm.com>
wrote:

Hi, all,
Quick reminder that the CAB call for September is this week Wednesday November 11th @ 8a PDT.
Please add any project updates to Agenda here: https://docs.google.com/document/d/1SCOlAquyUmNM-AQnekCOXiwhLs6gveTxAcduvDcW_xI/edit#heading=h.o44xhgvum2we
If you have something else to share, please also add an entry at the end.
Best,
Chip, James, and Max


CF CAB call for November is this Wednesday Nov. 11th, 2015

Michael Maximilien
 

Hi, all,

Quick reminder that the CAB call for September is this week Wednesday November 11th @ 8a PDT.

Please add any project updates to Agenda here: https://docs.google.com/document/d/1SCOlAquyUmNM-AQnekCOXiwhLs6gveTxAcduvDcW_xI/edit#heading=h.o44xhgvum2we

If you have something else to share, please also add an entry at the end.

Best,

Chip, James, and Max


Re: Immutability of applications

Dieu Cao <dcao@...>
 

Hi John,

I had been thinking about this a bit more, and I think it would be
reasonable to consider adding an end point that is similar to the /stats
end point, that queries Diego, to get actual state on environment variables
and as part of that VCAP_SERVICES for the running app instances.
Thoughts on that approach? Would that help address the problem?

-Dieu

On Wed, Nov 4, 2015 at 9:03 AM, john mcteague <john.mcteague(a)gmail.com>
wrote:

I had this conversation with a few different people during the berlin
summit and promised one of them I would repeat it on the mailing list today
to get further feedback.

Today, once we push an application, the droplet is immutable. It doesnt
change until you push the application again or restage. I believe the
entire container could change without a new push if you upgrade the rootfs
and restart all the apps (which the CF operator would do).

However, the environment vars and service bindings can be changed on an
application but they would not take affect until the next restart.The CF
API would report these changes as active when you run *cf env *or *cf
services. *There is no distinction between desired state and current
state when using the API.

To me this is a significant gap as we cannot necessarily get a true view
of the world (i call cf set-env but dont restart the app, how do I know
from the API what value of that env var my app is using).

How are people addressing this in their own environments and is it
something that the core API team should be considering (I ask the latter
publicly even though I asked Dieu during the summit :) ).

John


Re: Cloud Controller - s3 encryption for droplets

Dieu Cao <dcao@...>
 

I don't believe this is a bosh feature. Checked with Dmitriy and he
confirms.
Could you share any links that indicate that bosh supports that?
We could look to improve the documentation in that area to be more clear.

-Dieu

On Mon, Nov 9, 2015 at 3:35 PM, William C Penrod <wcpenrod(a)gmail.com> wrote:

The blobstore properties in bosh allows you to enter an encryption key to
encrypt the data before it is pushed up to an s3 bucket. In the cloud
foundry deploy, can this encryption property be added for the s3 connection
in the cloud controller for droplets?


Re: Source IP ACLs

Dieu Cao <dcao@...>
 

For apps/services hosted on the system domain that get their route by
publishing to nats or the routing-api, I believe it only requires
registering the route with the additional route_service_url. See the
routing-api-cli [1] for example usage.

As for api, login, etc, many of these components are now using a shared
route registrar job. I could imagine that job being extended to take a
configurable route_service_url.

As an FYI, I've been considering using route services for cloud controller
for rate limiting in the future.

[1] https://github.com/cloudfoundry-incubator/routing-api-cli

On Mon, Nov 9, 2015 at 8:56 PM, Carlo Alberto Ferraris <
carlo.ferraris(a)rakuten.com> wrote:

Our use case include restricting access to some hostnames on the system
domain (e.g. api/login/etc.) as well as others that we may add externally.
From a cursory read of the proposed spec it sounds like route services are
designed to be bound to apps running in CF, so IIUC it wouldn't be possible
to deny/allow connections to non-app endpoints.


Re: Source IP ACLs

Carlo Alberto Ferraris
 

Our use case include restricting access to some hostnames on the system domain (e.g. api/login/etc.) as well as others that we may add externally. From a cursory read of the proposed spec it sounds like route services are designed to be bound to apps running in CF, so IIUC it wouldn't be possible to deny/allow connections to non-app endpoints.


Re: Concourse OAuth on runtime.ci.cf-app.com

Christopher Piraino <cpiraino@...>
 

Woops, thought I was responding to Gabe. Disregard. :)

On Mon, Nov 9, 2015 at 5:29 PM, Christopher Piraino <cpiraino(a)pivotal.io>
wrote:

The routing team would like access! I see the OAuth is through Github, do
you need github handles or can you add teams?

- Chris

On Mon, Nov 9, 2015 at 5:05 PM, Gabriel Rosenhouse <grosenhouse(a)pivotal.io
wrote:
Hi all,

We've enabled OAuth login to https://runtime.ci.cf-app.com

Currently this is in addition to basic auth. We plan to eventually
retire basic auth.

The MEGA and CAPI teams have access via OAuth. Let us know if your team
also needs access.

Regards,

Gabe & Zak A.
MEGA team


Re: Concourse OAuth on runtime.ci.cf-app.com

Christopher Piraino <cpiraino@...>
 

The routing team would like access! I see the OAuth is through Github, do
you need github handles or can you add teams?

- Chris

On Mon, Nov 9, 2015 at 5:05 PM, Gabriel Rosenhouse <grosenhouse(a)pivotal.io>
wrote:

Hi all,

We've enabled OAuth login to https://runtime.ci.cf-app.com

Currently this is in addition to basic auth. We plan to eventually retire
basic auth.

The MEGA and CAPI teams have access via OAuth. Let us know if your team
also needs access.

Regards,

Gabe & Zak A.
MEGA team


Concourse OAuth on runtime.ci.cf-app.com

Gabriel Rosenhouse <grosenhouse@...>
 

Hi all,

We've enabled OAuth login to https://runtime.ci.cf-app.com

Currently this is in addition to basic auth. We plan to eventually retire
basic auth.

The MEGA and CAPI teams have access via OAuth. Let us know if your team
also needs access.

Regards,

Gabe & Zak A.
MEGA team


Re: [buildpacks] CF-built binaries: release timeline and beta testing

Shawn Nielsen
 

Mike,

Thanks for the quick reply.

My question is more around using an 'online' buildpack vs. 'offline'.
When I say 'online', I'm referring to the binaries being pulled from the s3
repo at staging time.

If we use the offline buildpack binaries, we are restricted to node
versions explicitly defined in the manifest files. This creates some
issues when projects want to do validation testing between one version and
the next or need to roll back for whatever reason. It also requires us to
manually update the buildpack every time there is a new release. We've
historically used the online buildpacks to prevent these types of issues,
but we'd like to better understand Pivotal's long term strategy here.

Going forward, what is Pivotal's strategy for online binaries that would be
used at staging time?

-Shawn

On Mon, Nov 9, 2015 at 4:03 PM, Mike Dalessio <mdalessio(a)pivotal.io> wrote:

Hi Shawn,

Thanks for asking.

The actual URL used is going to depend on what version of the buildpack
you're using. Currently master (v1.5.1) is only referncing CF-generated
binaries:

https://github.com/cloudfoundry/nodejs-buildpack/blob/master/manifest.yml

We removed references to s3pository.heroku.com in February, for v1.2.0 of
the buildpack.

Let me know if you need more context around this. Thanks!

-mike


On Mon, Nov 9, 2015 at 4:40 PM, Shawn Nielsen <sknielse(a)gmail.com> wrote:

James,

It looks like the online NodeJS buildpack is currently pointing to
Heroku's repo:
http://s3pository.heroku.com/node/...

Whereas the offline NodeJS buildpack manifest files seems to be pointing
to pivotal's repo:
https://pivotal-buildpacks.s3.amazonaws.com/concourse-binaries/node/...

Is there a reason for this discrepency?

I would think the online NodeJS buildpack should be pointing to the same
pivotal repo as the offline:
https://pivotal-buildpacks.s3.amazonaws.com/concourse-binaries/node/...


Let me know if you have any input here. Thanks,

-Shawn


On Fri, Jul 17, 2015 at 11:46 PM, James Bayer <jbayer(a)pivotal.io> wrote:

mike d is the best to answer, but i'll take a crack at it

On Fri, Jul 17, 2015 at 3:24 PM, Shawn Nielsen <sknielse(a)gmail.com>
wrote:

Two questions on these cf-built binaries:

1. From my understanding, the purpose of this is to compile the
binaries so they are optimized for the cf specific stacks (e.g.
cflinuxfs2 ) as opposed to something that was generically compiled
(e.g. to 64 bit trusty). Can you confirm this or expound upon this if
there are other reasons?
for some buildpacks, we have been relying on heroku binaries which is an
external dependency. we want the cf team to be in complete control of how
and when the binaries are built. this ensures cf can be in control of our
own destiny when for patching security issues or bugs. additionally it
means cf can take responsibility for how the binaries are compiled to
increase trust of the binary contents.



2. Are these binaries available in offline mode only or is there also
intent that they will be hosted, allowing us to consume them in an online
mode?
most/all cf buildpacks are available in online and offline mode as i
understand it. see:
https://github.com/cloudfoundry-incubator/buildpack-packager/blob/master/doc/disconnected_environments.md



Thanks,

-Shawn

On Mon, Jul 13, 2015 at 2:08 PM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:



On Mon, Jul 13, 2015 at 1:08 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Wed, Jul 8, 2015 at 1:55 PM, Mike Dalessio <mdalessio(a)pivotal.io>
wrote:

Hi all,

In the June CAB call, it was mentioned that the Buildpacks team was
working on generating CF-specific binaries to be packaged in the
buildpacks. I'm pleased to announce that the team is ready to start
shipping these binaries in the golang, nodejs, php, python, and ruby
buildpacks for the cflinuxfs2 stack.

*__We're planning to cut releases of these buildpacks with the CF*
*binaries on Monday, 20 July.__*

In the meantime, I'd like to ask the community to beta these
buildpacks and give us feedback.

We're successfully running [BRATs][1] (the buildpack runtime
acceptance tests) on these binaries, so we don't expect any issues; but
we'd love to hear if anyone experiences any issues deploying their apps.

Obviously, until we cut official releases, you should use judgement
when deciding where to use these beta buildpacks.

[1]: https://github.com/cloudfoundry/brats


*## Timeline, Versioning and Proposed Changes*

Unless we hear of any blocking issues, we'll cut official releases
of the go, node, php, python and ruby buildpacks on July 20th.

When we cut these releases, we'll be bumping the major version
number, and removing the `manifest-including-unsupported.yml` file from the
repository HEAD. I'd love to hear anyone's opinion on these changes as well.


*## How to deploy with the "beta" binary buildpacks*

Until we cut official releases, we are maintaining a git branch in
each buildpack repository, named `cf-binary-beta`, in which the manifest
points at our CF-specific binaries.

If your CF deployment has access to the public internet, you can
push your app and specify the github repo and branch with the `-b` option.

*__Go:__*

`cf push <appname> -b
https://github.com/cloudfoundry/go-buildpack#cf-binary-beta`
<https://github.com/cloudfoundry/go-buildpack#cf-binary-beta>

*__Node:__*

`cf push <appname> -b
https://github.com/cloudfoundry/nodejs-buildpack#cf-binary-beta`
<https://github.com/cloudfoundry/nodejs-buildpack#cf-binary-beta>

*__PHP:__*

`cf push <appname> -b
https://github.com/cloudfoundry/php-buildpack#cf-binary-beta`
<https://github.com/cloudfoundry/php-buildpack#cf-binary-beta>
Looks good mostly. One minor issue. It seems like the snmp
extension is not loading correctly. Looks to be missing a required shared
library.

2015-07-13T12:29:53.09-0400 [App/0] OUT 16:29:53 php-fpm |
[13-Jul-2015 16:29:53] NOTICE: PHP message: PHP Warning: PHP Startup:
Unable to load dynamic library
'/home/vcap/app/php/lib/php/extensions/no-debug-non-zts-20100525/snmp.so' -
libnetsnmp.so.30: cannot open shared object file: No such file or directory
in Unknown on line 0

That's with PHP 5.4.42, 5.5.26 and 5.6.10.
Good catch, I've created a story for this:
https://www.pivotaltracker.com/story/show/98980384





One other thing, which is not really an issue, but perhaps an
optimization. I noticed for the PHP extensions the bundle has both ".a"
and ".so" files for many of the extensions. The ".a" static libraries
should not be necessary, just the ".so" shared libraries. Seems like
removing them could save 8 to 9M. You could save another 25M by deleting
the bin/php-cgi file. You really only need bin/php for cmd line stuff and
sbin/php-fpm for web apps. Can't think of any reason you'd need / want to
do cgi. It's not a ton of space, 35M X 3 (each current version) + 35M X 3
(each of previous versions) - whatever compression will save. Just thought
I'd throw it out there though.

Another good catch, I've created a story to look into it:
https://www.pivotaltracker.com/story/show/98980692




Dan




*__Python:__*

`cf push <appname> -b
https://github.com/cloudfoundry/python-buildpack#cf-binary-beta`
<https://github.com/cloudfoundry/python-buildpack#cf-binary-beta>

*__Ruby:__*

`cf push <appname> -b
https://github.com/cloudfoundry/ruby-buildpack#cf-binary-beta`
<https://github.com/cloudfoundry/ruby-buildpack#cf-binary-beta>


If your CF deployment doesn't have access to the public internet and
you'd like to try these buildpacks, please reach out directly and we'll
figure out the best way to accommodate.


*## Tooling Details*

If you'd like to take a look at how we're currently building these
binaries, our tooling is open-sourced at:

https://github.com/cloudfoundry/binary-builder
Note that `binary-builder` uses the rootfs docker image to compile
each binary, which means that this tool can easily be extended to
essentially cross-compile **any** binary for a CF rootfs. We'd love
to hear your feedback on this, as well.


Thanks,
-mike



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

6761 - 6780 of 9425