Date   

Utilities PMC - 2015-09-09 Notes

Mike Dalessio
 

Hello CF Community,

Apologies for the lapse in sending timely Utilities PMC notes (see comments
below).

These notes are permanently available at:


https://github.com/cloudfoundry/pmc-notes/blob/master/Utilities/2015-09-09-utilities.md

-mike

---

# Utilities PMC Notes 2015-09-09

## Table of Contents

1. Update on "Offline" PMC Discussion
2. CLI
3. Java Utilities


## Update on "Offline" PMC Discussions

In the previous PMC meeting, held on 2015-07-28, the Utilities PMC agreed
to tentatively stop conducting regular synchronous meetings, and instead
move to an "offline" model where discussions and, if necessary, voting
would be conducted on the public cf-dev@ mailing list.

As part of this change, the PMC Lead (that's me!) committed to sending
regular status updates to the mailing list in lieu of meeting notes. In
this aspect, I failed over the last month, and I apologize for the
resulting lack of transparency.

Going forward, I'll return to a two-week cycle of email updates for this
PMC.

Please reach out to me or Chip if you have concerns over continuing the
"offline" model for the Utilities PMC.


## CLI

[CLI v6.12.3][] was released on August 31st, which notably includes many
community PRs and remove of codegangsta in preparation for an epic to
revamp `cf help`.

[CLI v6.12.3]: https://github.com/cloudfoundry/cli/releases/tag/v6.12.3

CLI has scheduled an inception on 2015-09-09 in San Francisco, Greg Oehmen
will be sending out details later this week.


## Java Utilities

Work continues on moving the Eclipse tools for Cloud Foundry to the Eclipse
Foundation. Immediate goal is inclusion in the Luna SR1 release.

The team is planning an inception for "v2" of `cf-java-client`. Immediate
goals will include support for CCv3 and to improve upon decisions made
during the initial implementation. We expect the inception planning process
to take some time, while some high-level design work is done to build
abstractions on top of a large number of API endpoints.


Re: UAA restart invalidates a valid token

Filip Hanik
 

After the fix, having override: true should not revoke the tokens. If it
still does, then it's a bug and we would like to know. thanks

On Wed, Sep 9, 2015 at 8:37 AM, Kayode Odeyemi <dreyemi(a)gmail.com> wrote:

Awesome.

option 2 is definitely the cause of the problem.

Thank you very much.

On Wed, Sep 9, 2015 at 4:32 PM, Filip Hanik <fhanik(a)pivotal.io> wrote:

We introduced a feature called 'revokable tokens'. A token would
automatically be revoked if a client changed it's secret. All tokens issued
previously would be automatically revoked.

In earlier versions of the UAA, if you have clients in your manifest, and
override flag set to true, even though the secret didn't change in the
manifest, the hashed secret was regenerated and thus this would expire all
the tokens.

you have a couple of different options
1. Update your UAA - this was fixed in
https://www.pivotaltracker.com/n/projects/997278/stories/97682912
2. Set override to false for your boot strapped clients and users



On Wed, Sep 9, 2015 at 8:22 AM, Kayode Odeyemi <dreyemi(a)gmail.com> wrote:

Hi,

What could cause a valid token to become invalid on UAA restart?

I've noticed this overtime, that a token (of client_credentials grant
type) which has a validity of 315360000 and has been used for
authentication and authorization of users and resource servers, suddenly
returns invalid_token when validated after a UAA restart.

{
"error": "invalid_token",
"error_description":
"eyJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI2M2JlZjY0Yi0yZDgzLTRhOTEtOGRlOS00NjI1YWM1N2NiNjMiLCJzdWIiOiJ1c2VyYWNjb3VudCIsImF1dGhvcml0aWVzIjpbImNsaWVudHMucmVhZCIsInNjaW0udXNlcmlkcyIsImNsaWVudHMuc2VjcmV0IiwidWFhLnJlc291cmNlIiwic2NpbS5tZSIsIm9wZW5pZCIsInVhYS5hZG1pbiIsInNjaW0ucmVhZCIsInBhc3N3b3JkLndyaXRlIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwiY2xvdWRfY29udHJvbGxlci53cml0ZSIsIm9hdXRoLmFwcHJvdmFscyIsImNsaWVudHMud3JpdGUiLCJzY2ltLndyaXRlIl0sInNjb3BlIjpbImNsaWVudHMucmVhZCIsInNjaW0udXNlcmlkcyIsImNsaWVudHMuc2VjcmV0IiwidWFhLnJlc291cmNlIiwic2NpbS5tZSIsIm9wZW5pZCIsInVhYS5hZG1pbiIsInNjaW0ucmVhZCIsInBhc3N3b3JkLndyaXRlIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwiY2xvdWRfY29udHJvbGxlci53cml0ZSIsIm9hdXRoLmFwcHJvdmFscyIsImNsaWVudHMud3JpdGUiLCJzY2ltLndyaXRlIl0sImNsaWVudF9pZCI6InVzZXJhY2NvdW50IiwiY2lkIjoidXNlcmFjY291bnQiLCJhenAiOiJ1c2VyYWNjb3VudCIsImdyYW50X3R5cGUiOiJjbGllbnRfY3JlZGVudGlhbHMiLCJyZXZfc2lnIjoiNWQ4Yjg5NGUiLCJpYXQiOjE0NDEzNTk0NzUsImV4cCI6MTc1NjcxOTQ3NSwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwL3VhYS9vYXV0aC90b2tlbiIsInppZCI6InVhYSIsImF1ZCI6WyJ1c2VyYWNjb3VudCIsImNsaWVudHMiLCJzY2ltIiwidWFhIiwib3BlbmlkIiwicGFzc3dvcmQiLCJjbG91ZF9jb250cm9sbGVyIiwib2F1dGgiXX0.qnOvyxBNKkDADZ2ODyQfZ98nj7cqoSGMIouduERU3Vg"
}

Any ideas please?


Re: UAA restart invalidates a valid token

Paul Bakare
 

Awesome.

option 2 is definitely the cause of the problem.

Thank you very much.

On Wed, Sep 9, 2015 at 4:32 PM, Filip Hanik <fhanik(a)pivotal.io> wrote:

We introduced a feature called 'revokable tokens'. A token would
automatically be revoked if a client changed it's secret. All tokens issued
previously would be automatically revoked.

In earlier versions of the UAA, if you have clients in your manifest, and
override flag set to true, even though the secret didn't change in the
manifest, the hashed secret was regenerated and thus this would expire all
the tokens.

you have a couple of different options
1. Update your UAA - this was fixed in
https://www.pivotaltracker.com/n/projects/997278/stories/97682912
2. Set override to false for your boot strapped clients and users



On Wed, Sep 9, 2015 at 8:22 AM, Kayode Odeyemi <dreyemi(a)gmail.com> wrote:

Hi,

What could cause a valid token to become invalid on UAA restart?

I've noticed this overtime, that a token (of client_credentials grant
type) which has a validity of 315360000 and has been used for
authentication and authorization of users and resource servers, suddenly
returns invalid_token when validated after a UAA restart.

{
"error": "invalid_token",
"error_description":
"eyJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI2M2JlZjY0Yi0yZDgzLTRhOTEtOGRlOS00NjI1YWM1N2NiNjMiLCJzdWIiOiJ1c2VyYWNjb3VudCIsImF1dGhvcml0aWVzIjpbImNsaWVudHMucmVhZCIsInNjaW0udXNlcmlkcyIsImNsaWVudHMuc2VjcmV0IiwidWFhLnJlc291cmNlIiwic2NpbS5tZSIsIm9wZW5pZCIsInVhYS5hZG1pbiIsInNjaW0ucmVhZCIsInBhc3N3b3JkLndyaXRlIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwiY2xvdWRfY29udHJvbGxlci53cml0ZSIsIm9hdXRoLmFwcHJvdmFscyIsImNsaWVudHMud3JpdGUiLCJzY2ltLndyaXRlIl0sInNjb3BlIjpbImNsaWVudHMucmVhZCIsInNjaW0udXNlcmlkcyIsImNsaWVudHMuc2VjcmV0IiwidWFhLnJlc291cmNlIiwic2NpbS5tZSIsIm9wZW5pZCIsInVhYS5hZG1pbiIsInNjaW0ucmVhZCIsInBhc3N3b3JkLndyaXRlIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwiY2xvdWRfY29udHJvbGxlci53cml0ZSIsIm9hdXRoLmFwcHJvdmFscyIsImNsaWVudHMud3JpdGUiLCJzY2ltLndyaXRlIl0sImNsaWVudF9pZCI6InVzZXJhY2NvdW50IiwiY2lkIjoidXNlcmFjY291bnQiLCJhenAiOiJ1c2VyYWNjb3VudCIsImdyYW50X3R5cGUiOiJjbGllbnRfY3JlZGVudGlhbHMiLCJyZXZfc2lnIjoiNWQ4Yjg5NGUiLCJpYXQiOjE0NDEzNTk0NzUsImV4cCI6MTc1NjcxOTQ3NSwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwL3VhYS9vYXV0aC90b2tlbiIsInppZCI6InVhYSIsImF1ZCI6WyJ1c2VyYWNjb3VudCIsImNsaWVudHMiLCJzY2ltIiwidWFhIiwib3BlbmlkIiwicGFzc3dvcmQiLCJjbG91ZF9jb250cm9sbGVyIiwib2F1dGgiXX0.qnOvyxBNKkDADZ2ODyQfZ98nj7cqoSGMIouduERU3Vg"
}

Any ideas please?


Buildpacks PMC - 2015-09-09 Notes

Mike Dalessio
 

Hello CF community,

Apologies for the lapse in sending timely Buildpacks PMC notes (see
comments below).

These notes are permanently available at:


https://github.com/cloudfoundry/pmc-notes/blob/master/Buildpacks/2015-09-09-buildpacks.md

-mike

-----

# Buildpacks PMC Notes as of 2015-09-09

## Table of Contents

1. Update on "Offline" PMC Discussion
2. Update on Stacks
3. Update on Buildpacks
1. General
2. go-buildpack
3. php-buildpack
4. ruby-buildpack
5. java-buildpack
4. Post-mortem on java-buildpack Certificate Expiration Incident


## Update on "Offline" PMC Discussions

In the previous PMC meeting, held on 2015-07-27, the Buildpacks PMC agreed
to tentatively stop conducting regular synchronous meetings, and instead
move to an "offline" model where discussions and, if necessary, voting
would be conducted on the public cf-dev@ mailing list.

As part of this change, the PMC Lead (that's me!) committed to sending
regular status updates to the mailing list in lieu of meeting notes. In
this aspect, I failed over the last month, and I apologize for the
resulting lack of transparency.

Going forward, I'll return to a two-week cycle of email updates for this
PMC.

Please reach out to me or Chip if you have concerns over continuing the
"offline" model for the Buildpacks PMC.


## Update on Stacks

[cflinuxfs2 v1.2.0][] through [cflinuxfs2 v1.7.0][] were released ([all
cflinuxfs2 releases][]), reflecting the team's focus on regular releases
and short turnaround times for CVEs.

[cflinuxfs2 v1.2.0]:
https://github.com/cloudfoundry/stacks/releases/tag/1.2.0
[cflinuxfs2 v1.7.0]:
https://github.com/cloudfoundry/stacks/releases/tag/1.7.0
[all cflinuxfs2 releases]: https://github.com/cloudfoundry/stacks/releases

One notable addition to the rootfs is the utility [jq][]. `jq` is commonly
vendored in custom buildpacks to parse CF environment variables (such as
`VCAP_SERVICES`). It's also used in the `go` buildpack to parse
`Godeps.json`. Adding `jq` to the rootfs is intended for use by future
buildpacks, to obviate the need to vendor a binary version of it.

[jq]: https://stedolan.github.io/jq/

Another notable change was introduced in [cflinuxfs2 v1.4.0][], which makes
it incompatible with Warden from cf212 and earlier. This change was
unfortunately necessary to support Garden and Diego. The release notes
contain instructions on how to modify Warden and cf-release if you need to
backport later rootfses into earlier cf-releases.

[cflinuxfs2 v1.4.0]:
https://github.com/cloudfoundry/stacks/releases/tag/1.4.0


## Update on Buildpacks

### General

Research continues on how to improve the state of extensibility for
Buildpacks. Tracker stories and their results are available in the
["architecture" epic][].

["architecture" epic]: https://www.pivotaltracker.com/epic/show/1898760


### go-buildpack

[go-buildpack v1.6.0][] and [go-buildpack v1.5.0][] were released. These
versions notably add support for golang 1.5.1, remove support for golang
1.1.x, update [godep][], and remove the vendored python interpreter.

[go-buildpack v1.6.0]:
https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.6.0
[go-buildpack v1.5.0]:
https://github.com/cloudfoundry/go-buildpack/releases/tag/v1.5.0
[godep]: https://github.com/tools/godep


### php-buildpack

[php-buildpack 4.1.2][] was released. This version notably makes a
non-backwards compatible change to composer detection, updates nginx to
1.9.3, and bumps the supported PHP versions to 5.6.12, 5.5.28 and 5.4.44.

[php-buildpack 4.1.2]:
https://github.com/cloudfoundry/php-buildpack/releases/tag/v4.1.2

We expect to release 4.1.3 this week with further updates to supported PHP
versions and nginx version.

A short survey was sent to cf-dev@ this week to evaluate whether, and how,
to support HHVM going forward. Please respond if you're a PHP user.
([survey link][])

[survey link]:
https://docs.google.com/forms/d/1WBupympWFRMQnoGZAgQLKmUZugreVldj3xDhyn9kpWM/viewform?usp=send_form

Finally, a note that PHP 5.4.x will reach End Of Life on 2015-09-14 (that's
next week). Beginning with php-buildpack v4.1.3, deprecation warnings will
be emitted upon staging apps that use PHP 5.4. We intend to remove PHP
5.4.x support from the php-buildpack within a month after EOL.


### ruby-buildpack

[ruby-buildpack v1.6.2][] through [ruby-buildpack v1.6.7][] were released.
Notably, these versions add support for JRuby 9.0.1.0 and 1.7.22, Ruby
2.2.3, 2.1.7 and 2.0.0-p647, openjdk 1.8.0_51, and sets the default Ruby
version to 2.2.3.

[ruby-buildpack v1.6.7]:
https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.6.7
[ruby-buildpack v1.6.2]:
https://github.com/cloudfoundry/ruby-buildpack/releases/tag/v1.6.2



### java-buildpack

[java-buildpack v3.1.1][] was released on July 30th, which updates
dependencies.

The java-buildpack team has also committed changes to `master` to add
zero-touch support for Safenet Luna HSMs, and to make a number of
improvements to the Java memory calculator.

[java-buildpack v3.1.1]:
https://github.com/cloudfoundry/java-buildpack/releases/tag/v3.1.1


## Post-mortem on java-buildpack Certificate Expiration Incident

On 2015-08-31, between 1300 UTC and 1554 UTC, applications failed to stage
that rely on an "online" (or "uncached") version of the java-buildpack, or
an "online" version of the ruby-buildpack who are using JRuby.

The root cause of this incident was the expiration of the SSL certificate
for `download.run.pivotal.io`, which is a domain used to front the
artifacts downloaded by the java-buildpack.

Two aspects of this incident were focused on during a post-mortem on
2015-09-09, to which Chip and Sam were invited.

1. __The certificate was allowed to expire__
2. __An open-source buildpack is relying on a Pivotal-managed domain and
SSL certificate__

The first aspect resulted in Pivotal-internal action items to centralize
cert management and to require an escalation policy for each cert. These
are perhaps uninteresting to most readers, but I'm happy to provide more
detail on request.

The second aspect is being addressed in part by [this Tracker story][] to
move to a Foundation-managed domain and cert.

[this Tracker story]:
https://www.pivotaltracker.com/n/projects/788065/stories/102935172

Ben Hale and I will be working with Chip and the Foundation to remove the
dependency on this domain as quickly as we can; and Pivotal is committing
to maintaining that domain and cert so existing buildpacks will continue to
work until users are migrated to updated versions.

One open question that was raised during the post-mortem: How can we better
communicate urgent issues like this to the CF community? The cf-dev@ list
tends to be a bit noisy, and isn't commonly used for urgent communications.
If anyone has ideas for how we can better respond as a community, I'd love
to hear your ideas.


Re: UAA restart invalidates a valid token

Filip Hanik
 

We introduced a feature called 'revokable tokens'. A token would
automatically be revoked if a client changed it's secret. All tokens issued
previously would be automatically revoked.

In earlier versions of the UAA, if you have clients in your manifest, and
override flag set to true, even though the secret didn't change in the
manifest, the hashed secret was regenerated and thus this would expire all
the tokens.

you have a couple of different options
1. Update your UAA - this was fixed in
https://www.pivotaltracker.com/n/projects/997278/stories/97682912
2. Set override to false for your boot strapped clients and users

On Wed, Sep 9, 2015 at 8:22 AM, Kayode Odeyemi <dreyemi(a)gmail.com> wrote:

Hi,

What could cause a valid token to become invalid on UAA restart?

I've noticed this overtime, that a token (of client_credentials grant
type) which has a validity of 315360000 and has been used for
authentication and authorization of users and resource servers, suddenly
returns invalid_token when validated after a UAA restart.

{
"error": "invalid_token",
"error_description":
"eyJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI2M2JlZjY0Yi0yZDgzLTRhOTEtOGRlOS00NjI1YWM1N2NiNjMiLCJzdWIiOiJ1c2VyYWNjb3VudCIsImF1dGhvcml0aWVzIjpbImNsaWVudHMucmVhZCIsInNjaW0udXNlcmlkcyIsImNsaWVudHMuc2VjcmV0IiwidWFhLnJlc291cmNlIiwic2NpbS5tZSIsIm9wZW5pZCIsInVhYS5hZG1pbiIsInNjaW0ucmVhZCIsInBhc3N3b3JkLndyaXRlIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwiY2xvdWRfY29udHJvbGxlci53cml0ZSIsIm9hdXRoLmFwcHJvdmFscyIsImNsaWVudHMud3JpdGUiLCJzY2ltLndyaXRlIl0sInNjb3BlIjpbImNsaWVudHMucmVhZCIsInNjaW0udXNlcmlkcyIsImNsaWVudHMuc2VjcmV0IiwidWFhLnJlc291cmNlIiwic2NpbS5tZSIsIm9wZW5pZCIsInVhYS5hZG1pbiIsInNjaW0ucmVhZCIsInBhc3N3b3JkLndyaXRlIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwiY2xvdWRfY29udHJvbGxlci53cml0ZSIsIm9hdXRoLmFwcHJvdmFscyIsImNsaWVudHMud3JpdGUiLCJzY2ltLndyaXRlIl0sImNsaWVudF9pZCI6InVzZXJhY2NvdW50IiwiY2lkIjoidXNlcmFjY291bnQiLCJhenAiOiJ1c2VyYWNjb3VudCIsImdyYW50X3R5cGUiOiJjbGllbnRfY3JlZGVudGlhbHMiLCJyZXZfc2lnIjoiNWQ4Yjg5NGUiLCJpYXQiOjE0NDEzNTk0NzUsImV4cCI6MTc1NjcxOTQ3NSwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwL3VhYS9vYXV0aC90b2tlbiIsInppZCI6InVhYSIsImF1ZCI6WyJ1c2VyYWNjb3VudCIsImNsaWVudHMiLCJzY2ltIiwidWFhIiwib3BlbmlkIiwicGFzc3dvcmQiLCJjbG91ZF9jb250cm9sbGVyIiwib2F1dGgiXX0.qnOvyxBNKkDADZ2ODyQfZ98nj7cqoSGMIouduERU3Vg"
}

Any ideas please?


UAA restart invalidates a valid token

Paul Bakare
 

Hi,

What could cause a valid token to become invalid on UAA restart?

I've noticed this overtime, that a token (of client_credentials grant type)
which has a validity of 315360000 and has been used for authentication and
authorization of users and resource servers, suddenly returns invalid_token
when validated after a UAA restart.

{
"error": "invalid_token",
"error_description":
"eyJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI2M2JlZjY0Yi0yZDgzLTRhOTEtOGRlOS00NjI1YWM1N2NiNjMiLCJzdWIiOiJ1c2VyYWNjb3VudCIsImF1dGhvcml0aWVzIjpbImNsaWVudHMucmVhZCIsInNjaW0udXNlcmlkcyIsImNsaWVudHMuc2VjcmV0IiwidWFhLnJlc291cmNlIiwic2NpbS5tZSIsIm9wZW5pZCIsInVhYS5hZG1pbiIsInNjaW0ucmVhZCIsInBhc3N3b3JkLndyaXRlIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwiY2xvdWRfY29udHJvbGxlci53cml0ZSIsIm9hdXRoLmFwcHJvdmFscyIsImNsaWVudHMud3JpdGUiLCJzY2ltLndyaXRlIl0sInNjb3BlIjpbImNsaWVudHMucmVhZCIsInNjaW0udXNlcmlkcyIsImNsaWVudHMuc2VjcmV0IiwidWFhLnJlc291cmNlIiwic2NpbS5tZSIsIm9wZW5pZCIsInVhYS5hZG1pbiIsInNjaW0ucmVhZCIsInBhc3N3b3JkLndyaXRlIiwiY2xvdWRfY29udHJvbGxlci5yZWFkIiwiY2xvdWRfY29udHJvbGxlci53cml0ZSIsIm9hdXRoLmFwcHJvdmFscyIsImNsaWVudHMud3JpdGUiLCJzY2ltLndyaXRlIl0sImNsaWVudF9pZCI6InVzZXJhY2NvdW50IiwiY2lkIjoidXNlcmFjY291bnQiLCJhenAiOiJ1c2VyYWNjb3VudCIsImdyYW50X3R5cGUiOiJjbGllbnRfY3JlZGVudGlhbHMiLCJyZXZfc2lnIjoiNWQ4Yjg5NGUiLCJpYXQiOjE0NDEzNTk0NzUsImV4cCI6MTc1NjcxOTQ3NSwiaXNzIjoiaHR0cDovL2xvY2FsaG9zdDo4MDgwL3VhYS9vYXV0aC90b2tlbiIsInppZCI6InVhYSIsImF1ZCI6WyJ1c2VyYWNjb3VudCIsImNsaWVudHMiLCJzY2ltIiwidWFhIiwib3BlbmlkIiwicGFzc3dvcmQiLCJjbG91ZF9jb250cm9sbGVyIiwib2F1dGgiXX0.qnOvyxBNKkDADZ2ODyQfZ98nj7cqoSGMIouduERU3Vg"
}

Any ideas please?


Re: Application only starts when a bogus service is attached

Fabien LEBRERE
 

Hi Amit,

Sorry, you right.
In fact, I'm an user of the CF' Ramon installation ;)

Daniel has solve my problems with the last link (JAVA_OPTS "-Djava.security.egd=file:///dev/urandom").

Cheers
Fabien


Re: When will dea be replaced by diego?

Layne Peng
 

Thanks Matthew. Very detail reply. :) It seems the all-in-one release won't come out soon.


Re: How to deploy a Web application using HTTPs

Juan Antonio Breña Moral <bren at juanantonio.info...>
 

Hi James,

Yes, you have reason, I returned to test:

https://nodejsssl.MY_IP.xip.io/

and I see the sreeen where Chrome advise the user about a NET::ERR_CERT_AUTHORITY_INVALID so, the node application is running:

https://raw.githubusercontent.com/jabrena/CloudFoundryLab/master/Node_HelloWorld_ssl/docs/firstScreen.png

but if you click to continue, I receive this message:

404 Not Found: Requested route ('nodejsssl.MY_IP.xip.io') does not exist.

My question is CF could fix this issue to deploy applications which it runs with https protocol.

Juan Antonio


Re: Placement Pools

Dieu Cao <dcao@...>
 

Yes, there are some differences in the approach from that document.

-Dieu

On Sat, Sep 5, 2015 at 2:21 PM, Noburou TANIGUCHI <dev(a)nota.m001.jp> wrote:

I'll take on getting the proposal for isolation groups shared with
cf-dev,
hopefully, in the next couple of weeks.
Great!

Is there any big change from

https://docs.google.com/document/d/1GNjQwGBh0BvfAYpX0LTUYn6h4oLz7v4P9pNy0xHZtMw/edit#
?


Dieu Cao wrote
Yes, I was just talking with Onsi and Mark Kropf about this yesterday and
plan to submit a talk on this with Mark Kropf for cf summit berlin.

I'll take on getting the proposal for isolation groups shared with
cf-dev,
hopefully, in the next couple of weeks.

-Dieu
CF CAPI PM

On Friday, September 4, 2015, James Bayer &lt;
jbayer@
&gt; wrote:

i believe dieu is working on this in preparate for cf summit in berlin.
dieu, did i understand that correctly?

On Thu, Sep 3, 2015 at 4:50 PM, Carlo Alberto Ferraris <
cafxx@
wrote:

James, Onsi,
we’re also looking forward to this, for we have some peculiar
infrastructure requirements.

Carlo

On Aug 29, 2015, at 2:51 AM, James Bayer &lt;
jbayer@
&gt; wrote:

we've been using a new term for the same concept we've previously
labeled
placement pools called "isolation groups".

onsi has been working on some documentation for what this may look like
and the requirements, but the work has not started. i believe onsi will
share something soon.

so today, the way to accomplish this need to place apps on specific
infrastructure is to use separate CF installations.

On Fri, Aug 28, 2015 at 8:50 AM, Matt Cholick &lt;
cholick@
&gt; wrote:

More than a year ago, there was some discussion and a proposal around
adding placement pools so cloud foundry admins could better target how
applications were placed on runners:

https://docs.google.com/document/d/1GNjQwGBh0BvfAYpX0LTUYn6h4oLz7v4P9pNy0xHZtMw/edit#

Did this work gain traction? I've looked through the release notes as
well as MEGA and CF Diego's public trackers and don't see stories for
this
work either done or planned, though it could also be that I'm just not
finding it.

My goal is to place canary apps in specifically Z1 or Z2, as well as
place some internally used apps that, for networking reasons, should
be
in
one zone or the other.

-Matt Cholick


--
Thank you,

James Bayer



--
Thank you,

James Bayer




-----
I'm not a ...
noburou taniguchi
--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-Placement-Pools-tp1394p1521.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: v3 cc api style guide feedback requested

Dieu Cao <dcao@...>
 

Hi Guillaume,

Just to ensure that it will be addressed, could you add it as github issues
on the repo? Hoping to do another pass at the remaining open issues later
this week.

-Dieu

On Mon, Sep 7, 2015 at 2:09 PM, Guillaume Berche <bercheg(a)gmail.com> wrote:

Thanks for sharing this great spec.

Not sure if you're preferring feedback other the mailing list of GH issue.
Let me know.

General feedback:

+1 for a formal schema for the v3 api as to ease automatic client
generations (api explorer, java sdk, go sdk...) (e.g. swagger format)
Automated tests on the formal schema may also help checking the style guide
is respected. https://www.pivotaltracker.com/story/show/99237980 seems to
only consider documentation benefits so far and not yet client generation
benefits (e.g. https://github.com/swagger-api/swagger-codegen
https://github.com/swagger-api/swagger-codegen/issues/325 )

Would be nice to clarify support for non ascii characters in query params,
such as support for IRI
https://en.wikipedia.org/wiki/Internationalized_resource_identifier as to
avoid mojibake bugs such as the one presumed in
https://github.com/cloudfoundry/cli/issues/560

Would be nice to consider supporting gzip encoding for the json payload
responses as to speed up responses over internet connections
('Accept-Encoding' header)

It general it may make sense to clarify supported HTTP headers (+1 for
etag/if-modified-since support suggested at
https://github.com/cloudfoundry/cc-api-v3-style-guide/issues/2 ).

https://github.com/cloudfoundry/cc-api-v3-style-guide#pagination
*"order_by: a field on the resource to order the collection by; each
collection may choose a subset of fields that it can be sorted by "*

Would be nice to illustrate/precise if multiple sort order can be
supported, e.g. order_by=-state,-created

https://github.com/cloudfoundry/cc-api-v3-style-guide#query-parameters
Precise character escaping on query param values e.g. containing comma:
filtering on name="a,b"


https://github.com/cloudfoundry/cc-api-v3-style-guide#pagination-of-related-resources

GET /v3/apps/:guid?include=space,organization

with pluralized resource name should be GET /v3/apps/:guid?include=space
*s*,organization*s*


https://github.com/cloudfoundry/cc-api-v3-style-guide#pagination-of-related-resources
would be nice to include an example of a pagination request on a related
resource inclusion request (e.g,

/v2/spaces/ab09cd29-9420-f021-g20d-123431420768?include=apps&*include_apps_order_by*=-state,-date)

https://github.com/cloudfoundry/cc-api-v3-style-guide#proposal
Would useful to consider I18N of user-facing messages. Cf related thread
for service broker error messages at
http://cf-dev.70369.x6.nabble.com/cf-dev-Announcing-Experimental-support-for-Asynchronous-Service-Operations-tp287p1471.html
May be the CC API could accept a "Accept-Language: zh_Hans" header and
try to return localized messages when available in the accepted locale.

Thanks,

Guillaume.

On Wed, Sep 2, 2015 at 6:44 PM, Zach Robinson <zrobinson(a)pivotal.io>
wrote:

Thanks James, I've just corrected the three issues you've noted so far


Re: When will dea be replaced by diego?

Guillaume Berche
 

Thanks a lot Matthew and Amit, the document can now be publicly
accessed/commented.

Guillaume.

On Tue, Sep 8, 2015 at 4:09 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Done, anyone with the link should be able to comment now.

Best,
Amit


On Tuesday, September 8, 2015, Matthew Sykes <matthew.sykes(a)gmail.com>
wrote:

Hi Guillaume. The proposal document was created by Amit and I had
assumed it was public. I'll try to make sure he sees this chain today so he
can address it. Sorry to send a unusable link.

On Tue, Sep 8, 2015 at 3:02 AM, Guillaume Berche <bercheg(a)gmail.com>
wrote:

Thanks Matthew for the additional details and pointers. It seems that
the deployment strategy proposal mentionned in [2] is lacking read/comment
permissions. Any chance to fix that ?

Guillaume.

On Tue, Sep 8, 2015 at 2:07 AM, Matthew Sykes <matthew.sykes(a)gmail.com>
wrote:

The notes you're pointing to were a straw man proposal; many of the
dates no longer seem relevant.

With that, I'm not in product management but, in my opinion, the
definition of "done" and "ready" are relative.

The current bar that the development team is focusing on is data and
API versioning. We feel it's necessary to maintain continuous operation
across deployments. In particular, we want to be sure that operators can
perform forward migration with minimal down time before it becomes the
default backend in production. We're currently referring to that target as
v 0.9.

That said, the current path towards that goal has us going to a single
API server Diego[1]. With this change in architecture, the scaling and
performance characteristics will probably change. While it's likely these
changes won't have measurable impact to smaller environments, it remains to
be seen what will happen with the larger deployments operated by public
providers. This is where the whole notion of "replacement" starts to get a
bit murky.

As for "merging into cf-release," again, I'm not product management
(James and Amit are in a better position to comment) but the current
direction appears to be to break down Cloud Foundry into a number of
smaller releases. We already have a cf-release, garden-release, and
diego-release as part of a diego deployment but there are others like an
etcd-release that the MEGA team is managing and a uaa-release that the
identity team have done. These are all pieces of a new deployment strategy
that was proposed[2] a few months ago.

Given that path, I don't know that diego-release will ever be merged
into cf-release; it's more likely that it will be stitched into the
"cf-deployment" described in that proposal.

So, to your question, the 0.9 release may be cut in September. That's
the first release that operators will be able to roll forward from without
downtime. If you want Diego to be the default backend without having to
mess with plugins and configuration, you can already do that today via
configuration[3].

[1]: https://github.com/onsi/migration-proposal
[2]:
https://docs.google.com/document/d/1Viga_TzUB2nLxN_ILqksmUiILM1hGhq7MBXxgLaUOkY/edit#heading=h.qam414rpl0xe
[3]:
https://github.com/cloudfoundry/cloud_controller_ng/blob/aea2a53b123dc5104c11eb53b81a09a4c4eaba55/bosh-templates/cloud_controller_api.yml.erb#L287

On Mon, Sep 7, 2015 at 2:08 PM, Layne Peng <layne.peng(a)emc.com> wrote:

I think what he ask is, when the Diego-release will merge to
cf-release. And also no need to install cf cli diego plugin, no need to
enabe-diego to your app, then start. For the
https://github.com/cloudfoundry-incubator/diego-design-notes/blob/master/migrating-to-diego.md#a-detailed-transition-timeline
. it is said to be mid-september, is it right?


--
Matthew Sykes
matthew.sykes(a)gmail.com

--
Matthew Sykes
matthew.sykes(a)gmail.com


Re: CAB September Call on 9/9/2015 @ 8a PDT

Amit Kumar Gupta
 

Hi all,

I will not be able to attend the CAB meeting tomorrow, but I have added my
notes to the agenda doc. MEGA has been/will be working on a bunch of
exciting things, and I welcome questions/comments via email, either through
the cf-dev mailing list or directly.

Best,
Amit, CF Release Integration team (MEGA) PM

On Tue, Sep 8, 2015 at 8:19 AM, Michael Maximilien <maxim(a)us.ibm.com> wrote:

Final reminder for the CAB call tomorrow. See you at Pivotal SF and talk
to you all then.

Best,

dr.max
ibm cloud labs
silicon valley, ca

Sent from my iPhone

On Sep 2, 2015, at 6:04 PM, Michael Maximilien <maxim(a)us.ibm.com> wrote:

Hi, all,

Quick reminder that the CAB call for September is next week Wednesday
September 9th @ 8a PDT.

Please add any project updates to Agenda here:
https://docs.google.com/document/d/1SCOlAquyUmNM-AQnekCOXiwhLs6gveTxAcduvDcW_xI/edit#heading=h.o44xhgvum2we

If you have something else to share, please also add an entry at the end.

Best,

Chip, James, and Max

PS: Dr.Nic this is one week in advance, so no excuses ;) phone info listed
on agenda.
PPS: Have a great labor day weekend---if you are in the US.


Re: So many hard-coded dropsonde destinations to metrons

Noburou TANIGUCHI
 

We're happy to see it.
Thanks a lot, Warren.


Warren Fernandes wrote
The LAMB team added a chore to discuss how we can better manage a
dropsonde_incoming_port on the metron_agent over here
https://www.pivotaltracker.com/story/show/102935222

We'll update this thread once we decide how to proceed.




-----
I'm not a ...
noburou taniguchi
--
View this message in context: http://cf-dev.70369.x6.nabble.com/So-many-hard-coded-dropsonde-destinations-to-metrons-tp1474p1565.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: Application only starts when a bogus service is attached

Amit Gupta
 

Hi all,

Ramon, Fabien, are you both working on the same problem or is Fabien's response to Daniel separate from Ramon's original question?

I couldn't understand Fabien's response. Did Daniel's help provide a satisfactory solution to the problem, or are you still using a workaround (whether it's binding a bogus service, or some other workaround)?

Thanks,
Amit, OSS Release Integration PM


Re: feedback request: extracting a common route-registrar job

Dieu Cao <dcao@...>
 

Agree with Zach, that for the CAPI team we would prefer to just wait for
the routing-api to be ready before prioritizing work to change the way CC
registers its routes.

Dieu

On Tuesday, September 8, 2015, Amit Gupta <agupta(a)pivotal.io> wrote:

Hey all,

The router already has some health check behaviour -- you can naively
register routes with it and it's smart about knowing whether to actually
route traffic or not. I'd like to understand what the specific use cases
are that would require a route-registration job to support additional
custom health check logic provided by the various colocated jobs.

Here's the set of things that are being done/need to be done for this
track of work:

1. extract route-registration process from UAA job in cf-release into its
own job in cf-release (note that the new uaa-release doesn't have this
process) [story done, needs acceptance]
2. use this route-registration job colocated with UAA in the cf manifests.
[story done, needs acceptance]
3. use this route-registration job colocated with CC, HM9k, and doppler in
the cf manifests. [stories in flight]
4. discover additional requirements for a generic route-registration job
to be useful [this email thread]
5. implement features in route-registration job to satisfy requirements
6. wait for consul to be declared stable, and enable the routing-api as a
default part of cf deployments [next couple final versions of cf-release
will validate consul]
7. all things doing route-registration should use routing-api
8. extract route-registration job out of cf-release (and put it where?) in
order to make it usable by other things like Gemfire, MySQL, Riak, etc.
9. discover more additional requirements for a generic route-registration
job to be useful
10. implement more features in route-registration job to satisfy
requirements
11. implement more features in routing-api to satisfy requirements

This isn't necessarily an ordered list, however we want to extract and
start using the route-registration stuff early to discover requirements,
and whether it's even a feasible idea, rather than waiting on consul and
routing-api to validate it. A nice additional consequence of doing this
extraction early is that once we want to switch over to routing-api, it
only needs to be done in one place.

Jens,

Here is a story in the Routing backlog to enable the routing-api to
support sticky-session route registrations:
https://www.pivotaltracker.com/story/show/100327146

Best,
Amit

On Tue, Sep 8, 2015 at 3:43 PM, Lyle Franklin <lfranklin(a)pivotal.io
<javascript:_e(%7B%7D,'cvml','lfranklin(a)pivotal.io');>> wrote:

The mysql and riak services register routes with this guy:
https://github.com/cloudfoundry-incubator/route-registrar. At a glance,
the option to run a healthcheck script to determine whether the service was
healthy would be our only real requirement in addition to the routing API
interaction. We'd love to be guinea pigs for any extracted library.

- Lyle

On Tue, Sep 8, 2015 at 1:38 PM, Zach Robinson <zrobinson(a)pivotal.io
<javascript:_e(%7B%7D,'cvml','zrobinson(a)pivotal.io');>> wrote:

i think it's a great idea, but I would wait until the routing api goes
live (waiting on consul stability) and use that tooling to create something
re-usable. I think the job for that could have some sort of health checky
thing where it would run a script with an agreed upon name to decide when
to register/unregister.

-Zach

On Tue, Sep 8, 2015 at 12:53 PM, Amit Gupta <agupta(a)pivotal.io
<javascript:_e(%7B%7D,'cvml','agupta(a)pivotal.io');>> wrote:

Hi all,

Several components within cf-release, as well as many jobs in different
releases, register a route with the gorouter:

- *doppler* registers the "doppler" and "loggregator" routes
- the *hm9000* API server registers the "hm9000" route
- *UAA* registers "uaa", "*.uaa", "login", and "*.login" routes
- *CC* registers the "api" route
- Many *service brokers* also register a route.

All these components register their routes in different ways. They
also all use the existing NATS flow, and will all need to switch their
implementations to use the routing API once that goes live and we start to
phase out NATS.

We have been working on extracting a route-registrar job which can be
colocated with other jobs and register routes on their behalf. Currently
it naively just always advertises the configured routes, and relies on the
gorouter's behaviour around knowing not to route requests to addresses that
aren't currently up.

One might require more sophisticated logic than this, however. For
example, a server may be "up" and theoretically capable of handling
requests, but not actually ready yet. Perhaps the router-registrar should
have some contract with its colocated jobs where those jobs can define a
health check script, and the route-registrar will only update the route
registration if the check succeeds.

Another requirement may exist around shutdown behaviour. Jobs may only
want to stop having its routes registered at a certain point in its drain
lifecycle.

*We would like feedback* from anyone maintaining a job or release that
does some sort of route registration to gather requirements that would be
desired of a generic route-registration component.

Thanks,
Amit, CF OSS Release Integration team


Relation between Network property in Resource pool block and Network property in Actual Job block

Ronak Banka
 

Hello All,

I have a resource pool , let say small_z1

- name: small_z1
network: cf1
stemcell: stemcell-xyz
cloud_properties:
instance_type: m1.small
availability_zone: zone1

and a Job , router having two networks assigned to it

- name: router
instances: 1
networks:
- name: router_internal
default: [dns, gateway]
static_ips:
- xy.xy.xy.xy
- name: router_external
static_ips:
- yz.yz.yz.yz
gateway: yy.yy.yy.yy
networks:
apps: router_internal
management: router_internal
resource_pool: small_z1

With these properties there are no issues anywhere.

what is the network property in resource pool responsible for, if the
created job networks and not linked to the one in pool??

Regards,
Ronak



--
View this message in context: http://cf-dev.70369.x6.nabble.com/Relation-between-Network-property-in-Resource-pool-block-and-Network-property-in-Actual-Job-block-tp1562.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: Any word on a large install version of CF on OpenStack?

Amit Kumar Gupta
 

Hi Michael,

What do you consider "large"? In principle, nothing is stopping you from
deploying a large cf deployment to your own OpenStack environment. When
you say "seeing a large install version of CF on OpenStack," whom would you
like to see produce this large install? There are several large
CF-on-OpenStack installs out there in the wild, but are you specifically
waiting on one from Stark & Wayne?

Best,
Amit

On Tue, Sep 8, 2015 at 2:55 PM, Michael Minges <mminges(a)ecsteam.com> wrote:

Curious whether there is any word on a possible time frame of seeing a
large install version of CF on OpenStack. I saw in the comments section of
a blog post (
https://blog.starkandwayne.com/2015/05/06/deploying-cloud-foundry-on-openstack-using-terraform/)
by Chris Weibel of Stark&Wayne that a cf-openstack-large.yml was in the
works but there was no set date for when that may be released to the
public. Currently we have a tiny deploy of CF on OpenStack for Dev but we
are looking to migrate our environment to a large install if this is in the
works and due out soon. Is this mailing list even the right place to start
this thread?

Thanks ahead for your time,

Michael Minges


Re: feedback request: extracting a common route-registrar job

Amit Kumar Gupta
 

Hey all,

The router already has some health check behaviour -- you can naively
register routes with it and it's smart about knowing whether to actually
route traffic or not. I'd like to understand what the specific use cases
are that would require a route-registration job to support additional
custom health check logic provided by the various colocated jobs.

Here's the set of things that are being done/need to be done for this track
of work:

1. extract route-registration process from UAA job in cf-release into its
own job in cf-release (note that the new uaa-release doesn't have this
process) [story done, needs acceptance]
2. use this route-registration job colocated with UAA in the cf manifests.
[story done, needs acceptance]
3. use this route-registration job colocated with CC, HM9k, and doppler in
the cf manifests. [stories in flight]
4. discover additional requirements for a generic route-registration job to
be useful [this email thread]
5. implement features in route-registration job to satisfy requirements
6. wait for consul to be declared stable, and enable the routing-api as a
default part of cf deployments [next couple final versions of cf-release
will validate consul]
7. all things doing route-registration should use routing-api
8. extract route-registration job out of cf-release (and put it where?) in
order to make it usable by other things like Gemfire, MySQL, Riak, etc.
9. discover more additional requirements for a generic route-registration
job to be useful
10. implement more features in route-registration job to satisfy
requirements
11. implement more features in routing-api to satisfy requirements

This isn't necessarily an ordered list, however we want to extract and
start using the route-registration stuff early to discover requirements,
and whether it's even a feasible idea, rather than waiting on consul and
routing-api to validate it. A nice additional consequence of doing this
extraction early is that once we want to switch over to routing-api, it
only needs to be done in one place.

Jens,

Here is a story in the Routing backlog to enable the routing-api to support
sticky-session route registrations:
https://www.pivotaltracker.com/story/show/100327146

Best,
Amit

On Tue, Sep 8, 2015 at 3:43 PM, Lyle Franklin <lfranklin(a)pivotal.io> wrote:

The mysql and riak services register routes with this guy:
https://github.com/cloudfoundry-incubator/route-registrar. At a glance,
the option to run a healthcheck script to determine whether the service was
healthy would be our only real requirement in addition to the routing API
interaction. We'd love to be guinea pigs for any extracted library.

- Lyle

On Tue, Sep 8, 2015 at 1:38 PM, Zach Robinson <zrobinson(a)pivotal.io>
wrote:

i think it's a great idea, but I would wait until the routing api goes
live (waiting on consul stability) and use that tooling to create something
re-usable. I think the job for that could have some sort of health checky
thing where it would run a script with an agreed upon name to decide when
to register/unregister.

-Zach

On Tue, Sep 8, 2015 at 12:53 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Hi all,

Several components within cf-release, as well as many jobs in different
releases, register a route with the gorouter:

- *doppler* registers the "doppler" and "loggregator" routes
- the *hm9000* API server registers the "hm9000" route
- *UAA* registers "uaa", "*.uaa", "login", and "*.login" routes
- *CC* registers the "api" route
- Many *service brokers* also register a route.

All these components register their routes in different ways. They also
all use the existing NATS flow, and will all need to switch their
implementations to use the routing API once that goes live and we start to
phase out NATS.

We have been working on extracting a route-registrar job which can be
colocated with other jobs and register routes on their behalf. Currently
it naively just always advertises the configured routes, and relies on the
gorouter's behaviour around knowing not to route requests to addresses that
aren't currently up.

One might require more sophisticated logic than this, however. For
example, a server may be "up" and theoretically capable of handling
requests, but not actually ready yet. Perhaps the router-registrar should
have some contract with its colocated jobs where those jobs can define a
health check script, and the route-registrar will only update the route
registration if the check succeeds.

Another requirement may exist around shutdown behaviour. Jobs may only
want to stop having its routes registered at a certain point in its drain
lifecycle.

*We would like feedback* from anyone maintaining a job or release that
does some sort of route registration to gather requirements that would be
desired of a generic route-registration component.

Thanks,
Amit, CF OSS Release Integration team


Re: feedback request: extracting a common route-registrar job

Lyle Franklin
 

The mysql and riak services register routes with this guy:
https://github.com/cloudfoundry-incubator/route-registrar. At a glance, the
option to run a healthcheck script to determine whether the service was
healthy would be our only real requirement in addition to the routing API
interaction. We'd love to be guinea pigs for any extracted library.

- Lyle

On Tue, Sep 8, 2015 at 1:38 PM, Zach Robinson <zrobinson(a)pivotal.io> wrote:

i think it's a great idea, but I would wait until the routing api goes
live (waiting on consul stability) and use that tooling to create something
re-usable. I think the job for that could have some sort of health checky
thing where it would run a script with an agreed upon name to decide when
to register/unregister.

-Zach

On Tue, Sep 8, 2015 at 12:53 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Hi all,

Several components within cf-release, as well as many jobs in different
releases, register a route with the gorouter:

- *doppler* registers the "doppler" and "loggregator" routes
- the *hm9000* API server registers the "hm9000" route
- *UAA* registers "uaa", "*.uaa", "login", and "*.login" routes
- *CC* registers the "api" route
- Many *service brokers* also register a route.

All these components register their routes in different ways. They also
all use the existing NATS flow, and will all need to switch their
implementations to use the routing API once that goes live and we start to
phase out NATS.

We have been working on extracting a route-registrar job which can be
colocated with other jobs and register routes on their behalf. Currently
it naively just always advertises the configured routes, and relies on the
gorouter's behaviour around knowing not to route requests to addresses that
aren't currently up.

One might require more sophisticated logic than this, however. For
example, a server may be "up" and theoretically capable of handling
requests, but not actually ready yet. Perhaps the router-registrar should
have some contract with its colocated jobs where those jobs can define a
health check script, and the route-registrar will only update the route
registration if the check succeeds.

Another requirement may exist around shutdown behaviour. Jobs may only
want to stop having its routes registered at a certain point in its drain
lifecycle.

*We would like feedback* from anyone maintaining a job or release that
does some sort of route registration to gather requirements that would be
desired of a generic route-registration component.

Thanks,
Amit, CF OSS Release Integration team