Date   

Re: Deploy CF on vsphere which managed by openstack

Yitao Jiang
 

Actually you need to build your own STEMCELL which is vmdk type as the
qcow2 mainly used for KVM hypervisor

On Tue, Aug 23, 2016 at 9:54 PM, Youzhi Zhu <zhuyouzhi03(a)gmail.com> wrote:

We use vsphere esxi as our hypervisor environment, and this environment is
managed by openstack,that is to say we use openstack to manage the
lifecycle of virtual machines and images and so on. Now I want to deploy CF
on this environment by bosh, so I should use bosh-openstack-cpi as the cpi,
but as the hypervisor is esxi, I think we need use vsphere-esxi-ubuntu-trusty-go_agent
as the stemcell, but it occured an error when creating stemcell as follow:

*main] 2016/08/23 18:30:26 ERROR - Command 'deploy' failed: creating
stemcell (bosh-vsphere-esxi-ubuntu-trusty-go_agent 3262.2): CPI
'create_stemcell' method responded with error:
CmdError{"type":"Bosh::Clouds::CloudError","message":"Root image is missing
from stemcell archive","ok_to_retry":false} *

I checked the openstack-cpi and found this error means missing the
"root.img" file. I also tried to use openstack-kvm-ubuntu-trusty-go_agent
as the stemcell, and images can be created successfully from it, but the
format is qcow2 which can not be used to create VM on vsphere esxi.

My questions:
Weather there is an existing solution for this situation? If not, what
should I do to meet this demand? Making our own stemcell or modify the
openstack cpi?
Thanks!

bosh version: 257.3
openstack-cpi-release:25
stemcell : 3262.2



--

Regards,

Yitao


Deploy CF on vsphere which managed by openstack

Youzhi Zhu
 

We use vsphere esxi as our hypervisor environment, and this environment is
managed by openstack,that is to say we use openstack to manage the
lifecycle of virtual machines and images and so on. Now I want to deploy CF
on this environment by bosh, so I should use bosh-openstack-cpi as the cpi,
but as the hypervisor is esxi, I think we need
use vsphere-esxi-ubuntu-trusty-go_agent as the stemcell, but it occured an
error when creating stemcell as follow:

*main] 2016/08/23 18:30:26 ERROR - Command 'deploy' failed: creating
stemcell (bosh-vsphere-esxi-ubuntu-trusty-go_agent 3262.2): CPI
'create_stemcell' method responded with error:
CmdError{"type":"Bosh::Clouds::CloudError","message":"Root image is missing
from stemcell archive","ok_to_retry":false} *

I checked the openstack-cpi and found this error means missing the
"root.img" file. I also tried to use openstack-kvm-ubuntu-trusty-go_agent
as the stemcell, and images can be created successfully from it, but the
format is qcow2 which can not be used to create VM on vsphere esxi.

My questions:
Weather there is an existing solution for this situation? If not, what
should I do to meet this demand? Making our own stemcell or modify the
openstack cpi?
Thanks!

bosh version: 257.3
openstack-cpi-release:25
stemcell : 3262.2


Re: DNS caching and forwarding in Cloud Foundry with Consul Agents

Hector Rivas Gandara
 

If you are able to configure consul-release to (a) not rewrite
/etc/resolv.conf and (b) not listen on port 53,
why would you need to specify which recursors consul hits, given that you
would probably never
send it queries outside consul's domain anyways, thus it would never try
any recursing?

Sorry I did not explain myself.

I did not mean that I want to implement all scenarios at the same time. I
meant that consul-release should be flexible enough to allow one to:

* Do not rewrite resolv.conf
* Bind to different port than 53.
* Allow override recursors with a explicit list of nameservers.

With these 3 options, one can implement any scenario without having to fork
consul-release.

I am telling you which minimum features think should be implemented in
consul-release, but that does not mean that somebody will use all these
options together.



On 22 August 2016 at 16:55, Amit Gupta <agupta(a)pivotal.io> wrote:

Hi Hector,

If you are able to configure consul-release to (a) not rewrite
/etc/resolv.conf and (b) not listen on port 53, then I assume you would run
your own thing like bind or dnsmasq there, and only forward specific
queries to the consul agent. In this case, why would you need to specify
which recursors consul hits, given that you would probably never send it
queries outside consul's domain anyways, thus it would never try any
recursing?

Best,
Amit

On Mon, Aug 22, 2016 at 5:21 AM, Hector Rivas Gandara <
hector.rivas.gandara(a)digital.cabinet-office.gov.uk> wrote:

This all sounds great. Sounds like you just want options to (1)
configure
which port Consul DNS binds to, and (2) toggle whether or not it makes
changes to /etc/resolv.conf, at minimum.
Yeah, that would be the minimum required.

Then, I would add:

(3) override the recursors to be used by consul.

With those 3x options, consul can be configured to support any
scenario without having to fork any release.

Is this something you'd want to configure Consul to do, or
would you configure your upstream component like dnsmasq or bind to
only hit
the Consul DNS server port when it receives a query for that search
domain?

Yes, that should be the pattern.

You also mentioned you don't like Consul's recursor behaviour, and want
to
"turn it off". Independently, other people have asked for Consul to be
able
to do more recursing, by being able to provide it a list of IPs to
recurse
to *in addition* to the ones it pulls out of /etc/resolv.conf. So I
see a
few use cases people might have:

(a) make Consul not do any recursing at all, and let the logic live
upstream
(in something like dnsmasq or bind) to try a different server when
Consul
isn't able to resolve an address
(b) doesn't matter at all what Consul recursing does, because you want
to
configure something like dnsmasq to only send queries to Consul when it
matches a certain domain, so no matter how Consul is configured to
recurse,
in practice this would never happen
(c) doesn't matter at all what Consul recursing does, because you want
to
configure Consul itself to only handle queries which match a certain
domain,
so no matter how Consul is configured to recurse, in practice this would
never happen
(d) be able to override the set of recursors Consul uses (instead of
what it
reads out of /etc/resolv.conf), so that you could set it to an empty
array
or whatever you like -- this is one way to generalize use case (a)
(e) be able to add a set of recursors to the ones Consul reads out of
/etc/resolv.conf, because the stuff in /etc/resolv.conf is determined
by the
BOSH Director, and not necessarily something you want to mess with, but
may
be something you want to add to, to get over limitations like "MAXNS 3".
See: https://www.pivotaltracker.com/story/show/121335409

If consul-release makes it possible to change the binding port and
toggle it
not to modify /etc/resolv.conf, then would you say your use cases falls
into
bucket (b)? If so, that's good because it requires fewer additional
changes
and also supports consul-release being able to support use case (e) as
well
at some point, without conflict.
Yes, I think that would be the case.

IMO I think it is enough to allow the cases b and d. That can be done
by implementing the features 1, 2 and 3 mention before.


(e) be able to add a set of recursors to the ones Consul reads out of
/etc/resolv.conf, because the stuff in /etc/resolv.conf is determined
by the
BOSH Director, and not necessarily something you want to mess with, but
may
be something you want to add to, to get over limitations like "MAXNS 3".
See: https://www.pivotaltracker.com/story/show/121335409
Rare case and suboptimal.

Be aware that in the current consul implementation, each recursor will
be queried in round robin, with a hardcoded timeout of 2 seconds. So,
if you have 4x recursors, you query might tike 8s to complete! and
given that the default timeout in linux is 5 seconds with 2x retries,
you will have to override that as well.

As I commented in a previous mail, the consul recursor logic is not
smart at all, compared with other agents like dnsmasq, pdnsd or bind.
If somebody wants to customise and improve the resolution logic, they
should look at those agents, not at consul.

Because that, I don't think it makes sense to put an option of "add
new recursors to the list". Instead limit it to optionally specify the
full list of DNS to be configured as recursors (and empty to disable).
"Add recursors" is a rare case and I think it is OK to have to a
little bit of duplication of defining the DNS nameservers twice.



--
Regards
Hector Rivas | GDS / Multi-Cloud PaaS

--
Regards
Hector Rivas | GDS / Multi-Cloud PaaS


Util for Concourse users

Daniel Jones
 

Hi all,

I knocked up a tiny utility that Concourse users might find handy.

It takes a YAML file of variables, and runs a command with those variables
in the environment. Handy if you use --load-vars-from when setting
pipelines, but then want to `fly execute` or run processes locally with the
same env vars as appear in those files.

https://github.com/EngineerBetter/yml2env

Regards,
Daniel Jones - CTO
+44 (0)79 8000 9153
@DanielJonesEB <https://twitter.com/DanielJonesEB>
*EngineerBetter* Ltd <http://www.engineerbetter.com> - UK Cloud Foundry
Specialists


Problem on binding service instance with an app

Kumaresh Babu
 

Hi,
I created haash service broker in out PCF environment. I created service instance for haash service and named as 'my-hash'.
I can't bind 'my-hash' service instance with an app. I am getting following exception.

Error: Server error, status code: 502, error code: 10001, message: The service broker rejected the request to https://haash-broker.*****com/v2/service_instances/187c8a7a-56f6-43b6-82a5-0912a86e95e4/service_bindings/c150de2a-f5a3-4b73-8013-d088537a11d5. Status Code: 400 Bad Request, Body: {"timestamp":1471833736776,"error":"Bad Request","status":400,"message":""}

***** - domain name. Kindly do needful.


Re: Is there a UAA API analog to the uaa.yml spring_profiles setting?

Eric Promislow
 

And finally I did a mapping of uaa.yml property names to config keys in the
API, and uaa is hitting my server. Problem solved.

On Mon, Aug 22, 2016 at 12:43 PM, Eric Promislow <eric.promislow(a)gmail.com>
wrote:

The file in the WAR file I'm using is called "ldap-groups-null.xml", but
changing that still didn't fix the problem.

There are a fair # of discrepancies between the property names in the
uaa.yml file and those in the published API. Should I go with the names in
uaa.yml, which I've seen working?

On Mon, Aug 22, 2016 at 12:26 PM, Eric Promislow <eric.promislow(a)gmail.com
wrote:
This log entry might point to the problem. Will investigate...

c-7] .... ERROR --- HomeController: Caused by:
org.springframework.beans.factory.BeanDefinitionStoreException:
IOException parsing XML document from URL [file:/usr/local/tomcat/webapp
s/ROOT/WEB-INF/classes/ldap/ldap-no-groups.xml]; nested exception is
java.io.FileNotFoundException: /usr/local/tomcat/webapps/ROOT
/WEB-INF/classes/ldap/ldap-no-groups.xml (No such file or directory)




On Mon, Aug 22, 2016 at 11:38 AM, Eric Promislow <
eric.promislow(a)gmail.com> wrote:

I'm trying to use the UAA API to add an LDAP server dynamically,
compared to specifying one in uaa.yml and restarting uaa.

In the YAML file I had to set spring_profiles to "postgresql,ldap"

I haven't found anything analogous in the UAA API, but ensuring that the
new ldap identity-provider is active isn't sufficient.

Any suggestions?

- Eric


Re: Is there a UAA API analog to the uaa.yml spring_profiles setting?

Eric Promislow
 

The file in the WAR file I'm using is called "ldap-groups-null.xml", but
changing that still didn't fix the problem.

There are a fair # of discrepancies between the property names in the
uaa.yml file and those in the published API. Should I go with the names in
uaa.yml, which I've seen working?

On Mon, Aug 22, 2016 at 12:26 PM, Eric Promislow <eric.promislow(a)gmail.com>
wrote:

This log entry might point to the problem. Will investigate...

c-7] .... ERROR --- HomeController: Caused by: org.springframework.beans.
factory.BeanDefinitionStoreException: IOException parsing XML document
from URL [file:/usr/local/tomcat/webapps/ROOT/WEB-INF/classes/ldap/ldap-no-groups.xml];
nested exception is java.io.FileNotFoundException:
/usr/local/tomcat/webapps/ROOT/WEB-INF/classes/ldap/ldap-no-groups.xml
(No such file or directory)




On Mon, Aug 22, 2016 at 11:38 AM, Eric Promislow <eric.promislow(a)gmail.com
wrote:
I'm trying to use the UAA API to add an LDAP server dynamically, compared
to specifying one in uaa.yml and restarting uaa.

In the YAML file I had to set spring_profiles to "postgresql,ldap"

I haven't found anything analogous in the UAA API, but ensuring that the
new ldap identity-provider is active isn't sufficient.

Any suggestions?

- Eric


Re: Is there a UAA API analog to the uaa.yml spring_profiles setting?

Eric Promislow
 

This log entry might point to the problem. Will investigate...

c-7] .... ERROR --- HomeController: Caused by:
org.springframework.beans.factory.BeanDefinitionStoreException: IOException
parsing XML document from URL
[file:/usr/local/tomcat/webapps/ROOT/WEB-INF/classes/ldap/ldap-no-groups.xml];
nested exception is java.io.FileNotFoundException:
/usr/local/tomcat/webapps/ROOT/WEB-INF/classes/ldap/ldap-no-groups.xml (No
such file or directory)



On Mon, Aug 22, 2016 at 11:38 AM, Eric Promislow <eric.promislow(a)gmail.com>
wrote:

I'm trying to use the UAA API to add an LDAP server dynamically, compared
to specifying one in uaa.yml and restarting uaa.

In the YAML file I had to set spring_profiles to "postgresql,ldap"

I haven't found anything analogous in the UAA API, but ensuring that the
new ldap identity-provider is active isn't sufficient.

Any suggestions?

- Eric


Is there a UAA API analog to the uaa.yml spring_profiles setting?

Eric Promislow
 

I'm trying to use the UAA API to add an LDAP server dynamically, compared
to specifying one in uaa.yml and restarting uaa.

In the YAML file I had to set spring_profiles to "postgresql,ldap"

I haven't found anything analogous in the UAA API, but ensuring that the
new ldap identity-provider is active isn't sufficient.

Any suggestions?

- Eric


Re: Cloud Foundry Java Client 2.0.0.RELEASE

Josh Long <starbuxman@...>
 

Congrats! This is a huge release! ^5 team on the amazing work!
On Mon, Aug 22, 2016 at 12:46 Daniel Jones <daniel.jones(a)engineerbetter.com>
wrote:

Congrats! It's awesome to have a decent CC API implementation that's
properly maintained.

Regards,
Daniel Jones - CTO
+44 (0)79 8000 9153
@DanielJonesEB <https://twitter.com/DanielJonesEB>
*EngineerBetter* Ltd <http://www.engineerbetter.com> - UK Cloud Foundry
Specialists

On Mon, Aug 22, 2016 at 5:41 PM, Ben Hale <bhale(a)pivotal.io> wrote:

I'm immensely pleased to announce Cloud Foundry Java Client
`2.0.0.RELEASE`! This release is the culmination of 54 weeks of work,
encompassing 10 milestones, two release candidates, and 2,344 commits[1].

Before the typical discussion about the release, I'd like to take a
moment to recognize everyone who contributed to the success of this effort:

* Steve Powell[2]
* Chris Frost[3]
* Benjamin Einaudi[4]
* Paul Harris[5]
* Glyn Normington[6]
* Lokesh Kumar N[7]
* Olivier Orand[8]
* Sebastien Bortolussi[9]
* Stephane Maldini[10]
* Ben Patterson[11]
* Mike Heath[12]
* Ramnivas Laddad[13]
* William Gautier[14]

It really makes me happy to see that more than half of the contributors
to the project are from outside of Pivotal. I see this as an indication
both that we're fulfilling a wide-spread need and that we've created an
environment that encourages external contribution. I encourage anyone else
interested in contributing to join this list over the coming months.

Next, I'd like to recognize the lead of Project Reactor[15] Stephane
Maldini, for his tireless efforts to help us deliver an incredible reactive
API. When we took the decision to head in this direction, Stephane stepped
up to teach the entire team the principles behind reactive programming.
This effort was non-trivial, entailing hundreds of hours on Screenhero[16]
often until after one in the morning for him. The last year has easily
been the greatest example of two projects evolving together, that I have
ever experienced. The amount of improvement Reactor and the Java Client
have driven in each other cannot be overestimated.

Finally I'd like to thank the teams that started building on the Java
Client, long before it was production ready. Projects like Spring Cloud
Data Flow[17] and Spring Cloud Spinnaker[18] have been invaluable tools for
driving usability and bug fixes.

And now onto the release!

* * *

This release was a complete re-write[19] of the Java Client with a focus
on:

* two cleanly delineated APIs; a `-client` API mapping to the REST calls
and an `-operations` API mapping to CLI
* implementing every single REST call exposed by all Cloud Foundry
components (more than 500 individual URIs across 4 components)
* an airtight separation between APIs and implementation; a Reactor-based
default is provided, and alternate implementations are possible
* a reactive API based on Project Reactor and interoperable with any
Reactive Streams-compatible[20] library

_For more detail on these goals, please see the 2.0.0.M1 Release
Notes[21]_

For detailed information on how to use the client, the README[22] is the
best place to start. Javadocs for the library can be found at
`cloudfoundry-client`[23], `cloudfoundry-client-reactor`[24],
`cloudfoundry-operations`[25], and `cloudfoundry-util`[26]. Since usage
requires a fair bit of knowledge about the workings of the Cloud Foundry
APIs themselves, check out the docs for the Cloud Controller V2[27]),
V3[28], Doppler[29], and the UAA[30].

While the Java Client implements a great many of the Cloud Foundry APIs
we didn't manage to complete them _all_. By our count we're about 80% the
way through and have delivered all of the commonly used endpoints. Over
the coming months we'll be continuing to deliver the missing
implementations until everything is complete. Beyond that, we're committed
to keeping the library up to date with Cloud Foundry's evolving API.

We welcome all feedback from the community and look forward to seeing you
open issues[31] and submit pull requests[32]. If you're looking for help
and discussion we're lurking in the Cloud Foundry Slack[33] during European
and North American working hours.

We hope that the new Java Client is useful for everyone and can't wait to
see what you build with it.


-Ben Hale
Cloud Foundry Java Experience


[1]:
https://github.com/cloudfoundry/cf-java-client/compare/dc8f8d5c8370de223992a0e4a024d5cff082a682...v2.0.0.RELEASE
[2]: https://github.com/cloudfoundry/cf-java-client/commits?author=Zteve
[3]:
https://github.com/cloudfoundry/cf-java-client/commits?author=cgfrost
[4]:
https://github.com/cloudfoundry/cf-java-client/commits?author=antechrestos
[5]:
https://github.com/cloudfoundry/cf-java-client/commits?author=twoseat
[6]: https://github.com/cloudfoundry/cf-java-client/commits?author=glyn
[7]:
https://github.com/cloudfoundry/cf-java-client/commits?author=LokeshN
[8]:
https://github.com/cloudfoundry/cf-java-client/commits?author=o-orand
[9]:
https://github.com/cloudfoundry/cf-java-client/commits?author=s-bortolussi
[10]:
https://github.com/cloudfoundry/cf-java-client/commits?author=smaldini
[11]:
https://github.com/cloudfoundry/cf-java-client/commits?author=benpatt
[12]:
https://github.com/cloudfoundry/cf-java-client/commits?author=mheath
[13]:
https://github.com/cloudfoundry/cf-java-client/commits?author=ramnivas
[14]:
https://github.com/cloudfoundry/cf-java-client/commits?author=WGautier
[15]: https://projectreactor.io
[16]: https://screenhero.com
[17]: https://github.com/spring-cloud/spring-cloud-dataflow
[18]: https://github.com/spring-cloud/spring-cloud-spinnaker
[19]:
https://github.com/cloudfoundry/cf-java-client/commit/dc8f8d5c8370de223992a0e4a024d5cff082a682
[20]: http://www.reactive-streams.org
[21]:
https://github.com/cloudfoundry/cf-java-client/releases/tag/v2.0.0.M1
[22]: https://github.com/cloudfoundry/cf-java-client/tree/v2.0.0.RELEASE
[23]:
http://cloudfoundry.github.io/cf-java-client/api/latest-release/cloudfoundry-client
[24]:
http://cloudfoundry.github.io/cf-java-client/api/latest-release/cloudfoundry-client-reactor
[25]:
http://cloudfoundry.github.io/cf-java-client/api/latest-release/cloudfoundry-operations
[26]:
http://cloudfoundry.github.io/cf-java-client/api/latest-release/cloudfoundry-util
[27]: http://apidocs.cloudfoundry.org/latest-release
[28]:
http://v3-apidocs.cloudfoundry.org/version/release-candidate/index.html
[29]:
https://github.com/cloudfoundry/loggregator/tree/develop/src/trafficcontroller#endpoints
[30]: http://docs.cloudfoundry.com/uaa
[31]: https://github.com/cloudfoundry/cf-java-client/issues
[32]: https://github.com/cloudfoundry/cf-java-client/pulls
[33]: https://cloudfoundry.slack.com/messages/java-client/


Re: Cloud Foundry Java Client 2.0.0.RELEASE

Daniel Jones
 

Congrats! It's awesome to have a decent CC API implementation that's
properly maintained.

Regards,
Daniel Jones - CTO
+44 (0)79 8000 9153
@DanielJonesEB <https://twitter.com/DanielJonesEB>
*EngineerBetter* Ltd <http://www.engineerbetter.com> - UK Cloud Foundry
Specialists

On Mon, Aug 22, 2016 at 5:41 PM, Ben Hale <bhale(a)pivotal.io> wrote:

I'm immensely pleased to announce Cloud Foundry Java Client
`2.0.0.RELEASE`! This release is the culmination of 54 weeks of work,
encompassing 10 milestones, two release candidates, and 2,344 commits[1].

Before the typical discussion about the release, I'd like to take a moment
to recognize everyone who contributed to the success of this effort:

* Steve Powell[2]
* Chris Frost[3]
* Benjamin Einaudi[4]
* Paul Harris[5]
* Glyn Normington[6]
* Lokesh Kumar N[7]
* Olivier Orand[8]
* Sebastien Bortolussi[9]
* Stephane Maldini[10]
* Ben Patterson[11]
* Mike Heath[12]
* Ramnivas Laddad[13]
* William Gautier[14]

It really makes me happy to see that more than half of the contributors to
the project are from outside of Pivotal. I see this as an indication both
that we're fulfilling a wide-spread need and that we've created an
environment that encourages external contribution. I encourage anyone else
interested in contributing to join this list over the coming months.

Next, I'd like to recognize the lead of Project Reactor[15] Stephane
Maldini, for his tireless efforts to help us deliver an incredible reactive
API. When we took the decision to head in this direction, Stephane stepped
up to teach the entire team the principles behind reactive programming.
This effort was non-trivial, entailing hundreds of hours on Screenhero[16]
often until after one in the morning for him. The last year has easily
been the greatest example of two projects evolving together, that I have
ever experienced. The amount of improvement Reactor and the Java Client
have driven in each other cannot be overestimated.

Finally I'd like to thank the teams that started building on the Java
Client, long before it was production ready. Projects like Spring Cloud
Data Flow[17] and Spring Cloud Spinnaker[18] have been invaluable tools for
driving usability and bug fixes.

And now onto the release!

* * *

This release was a complete re-write[19] of the Java Client with a focus
on:

* two cleanly delineated APIs; a `-client` API mapping to the REST calls
and an `-operations` API mapping to CLI
* implementing every single REST call exposed by all Cloud Foundry
components (more than 500 individual URIs across 4 components)
* an airtight separation between APIs and implementation; a Reactor-based
default is provided, and alternate implementations are possible
* a reactive API based on Project Reactor and interoperable with any
Reactive Streams-compatible[20] library

_For more detail on these goals, please see the 2.0.0.M1 Release Notes[21]_

For detailed information on how to use the client, the README[22] is the
best place to start. Javadocs for the library can be found at
`cloudfoundry-client`[23], `cloudfoundry-client-reactor`[24],
`cloudfoundry-operations`[25], and `cloudfoundry-util`[26]. Since usage
requires a fair bit of knowledge about the workings of the Cloud Foundry
APIs themselves, check out the docs for the Cloud Controller V2[27]),
V3[28], Doppler[29], and the UAA[30].

While the Java Client implements a great many of the Cloud Foundry APIs we
didn't manage to complete them _all_. By our count we're about 80% the way
through and have delivered all of the commonly used endpoints. Over the
coming months we'll be continuing to deliver the missing implementations
until everything is complete. Beyond that, we're committed to keeping the
library up to date with Cloud Foundry's evolving API.

We welcome all feedback from the community and look forward to seeing you
open issues[31] and submit pull requests[32]. If you're looking for help
and discussion we're lurking in the Cloud Foundry Slack[33] during European
and North American working hours.

We hope that the new Java Client is useful for everyone and can't wait to
see what you build with it.


-Ben Hale
Cloud Foundry Java Experience


[1]: https://github.com/cloudfoundry/cf-java-client/compare/
dc8f8d5c8370de223992a0e4a024d5cff082a682...v2.0.0.RELEASE
[2]: https://github.com/cloudfoundry/cf-java-client/commits?author=Zteve
[3]: https://github.com/cloudfoundry/cf-java-client/commits?author=cgfrost
[4]: https://github.com/cloudfoundry/cf-java-client/
commits?author=antechrestos
[5]: https://github.com/cloudfoundry/cf-java-client/commits?author=twoseat
[6]: https://github.com/cloudfoundry/cf-java-client/commits?author=glyn
[7]: https://github.com/cloudfoundry/cf-java-client/commits?author=LokeshN
[8]: https://github.com/cloudfoundry/cf-java-client/commits?author=o-orand
[9]: https://github.com/cloudfoundry/cf-java-client/
commits?author=s-bortolussi
[10]: https://github.com/cloudfoundry/cf-java-client/
commits?author=smaldini
[11]: https://github.com/cloudfoundry/cf-java-client/
commits?author=benpatt
[12]: https://github.com/cloudfoundry/cf-java-client/commits?author=mheath
[13]: https://github.com/cloudfoundry/cf-java-client/
commits?author=ramnivas
[14]: https://github.com/cloudfoundry/cf-java-client/
commits?author=WGautier
[15]: https://projectreactor.io
[16]: https://screenhero.com
[17]: https://github.com/spring-cloud/spring-cloud-dataflow
[18]: https://github.com/spring-cloud/spring-cloud-spinnaker
[19]: https://github.com/cloudfoundry/cf-java-client/commit/
dc8f8d5c8370de223992a0e4a024d5cff082a682
[20]: http://www.reactive-streams.org
[21]: https://github.com/cloudfoundry/cf-java-client/
releases/tag/v2.0.0.M1
[22]: https://github.com/cloudfoundry/cf-java-client/tree/v2.0.0.RELEASE
[23]: http://cloudfoundry.github.io/cf-java-client/api/latest-
release/cloudfoundry-client
[24]: http://cloudfoundry.github.io/cf-java-client/api/latest-
release/cloudfoundry-client-reactor
[25]: http://cloudfoundry.github.io/cf-java-client/api/latest-
release/cloudfoundry-operations
[26]: http://cloudfoundry.github.io/cf-java-client/api/latest-
release/cloudfoundry-util
[27]: http://apidocs.cloudfoundry.org/latest-release
[28]: http://v3-apidocs.cloudfoundry.org/version/
release-candidate/index.html
[29]: https://github.com/cloudfoundry/loggregator/tree/
develop/src/trafficcontroller#endpoints
[30]: http://docs.cloudfoundry.com/uaa
[31]: https://github.com/cloudfoundry/cf-java-client/issues
[32]: https://github.com/cloudfoundry/cf-java-client/pulls
[33]: https://cloudfoundry.slack.com/messages/java-client/


Cloud Foundry Java Client 2.0.0.RELEASE

Ben Hale <bhale@...>
 

I'm immensely pleased to announce Cloud Foundry Java Client `2.0.0.RELEASE`! This release is the culmination of 54 weeks of work, encompassing 10 milestones, two release candidates, and 2,344 commits[1].

Before the typical discussion about the release, I'd like to take a moment to recognize everyone who contributed to the success of this effort:

* Steve Powell[2]
* Chris Frost[3]
* Benjamin Einaudi[4]
* Paul Harris[5]
* Glyn Normington[6]
* Lokesh Kumar N[7]
* Olivier Orand[8]
* Sebastien Bortolussi[9]
* Stephane Maldini[10]
* Ben Patterson[11]
* Mike Heath[12]
* Ramnivas Laddad[13]
* William Gautier[14]

It really makes me happy to see that more than half of the contributors to the project are from outside of Pivotal. I see this as an indication both that we're fulfilling a wide-spread need and that we've created an environment that encourages external contribution. I encourage anyone else interested in contributing to join this list over the coming months.

Next, I'd like to recognize the lead of Project Reactor[15] Stephane Maldini, for his tireless efforts to help us deliver an incredible reactive API. When we took the decision to head in this direction, Stephane stepped up to teach the entire team the principles behind reactive programming. This effort was non-trivial, entailing hundreds of hours on Screenhero[16] often until after one in the morning for him. The last year has easily been the greatest example of two projects evolving together, that I have ever experienced. The amount of improvement Reactor and the Java Client have driven in each other cannot be overestimated.

Finally I'd like to thank the teams that started building on the Java Client, long before it was production ready. Projects like Spring Cloud Data Flow[17] and Spring Cloud Spinnaker[18] have been invaluable tools for driving usability and bug fixes.

And now onto the release!

* * *

This release was a complete re-write[19] of the Java Client with a focus on:

* two cleanly delineated APIs; a `-client` API mapping to the REST calls and an `-operations` API mapping to CLI
* implementing every single REST call exposed by all Cloud Foundry components (more than 500 individual URIs across 4 components)
* an airtight separation between APIs and implementation; a Reactor-based default is provided, and alternate implementations are possible
* a reactive API based on Project Reactor and interoperable with any Reactive Streams-compatible[20] library

_For more detail on these goals, please see the 2.0.0.M1 Release Notes[21]_

For detailed information on how to use the client, the README[22] is the best place to start. Javadocs for the library can be found at `cloudfoundry-client`[23], `cloudfoundry-client-reactor`[24], `cloudfoundry-operations`[25], and `cloudfoundry-util`[26]. Since usage requires a fair bit of knowledge about the workings of the Cloud Foundry APIs themselves, check out the docs for the Cloud Controller V2[27]), V3[28], Doppler[29], and the UAA[30].

While the Java Client implements a great many of the Cloud Foundry APIs we didn't manage to complete them _all_. By our count we're about 80% the way through and have delivered all of the commonly used endpoints. Over the coming months we'll be continuing to deliver the missing implementations until everything is complete. Beyond that, we're committed to keeping the library up to date with Cloud Foundry's evolving API.

We welcome all feedback from the community and look forward to seeing you open issues[31] and submit pull requests[32]. If you're looking for help and discussion we're lurking in the Cloud Foundry Slack[33] during European and North American working hours.

We hope that the new Java Client is useful for everyone and can't wait to see what you build with it.


-Ben Hale
Cloud Foundry Java Experience


[1]: https://github.com/cloudfoundry/cf-java-client/compare/dc8f8d5c8370de223992a0e4a024d5cff082a682...v2.0.0.RELEASE
[2]: https://github.com/cloudfoundry/cf-java-client/commits?author=Zteve
[3]: https://github.com/cloudfoundry/cf-java-client/commits?author=cgfrost
[4]: https://github.com/cloudfoundry/cf-java-client/commits?author=antechrestos
[5]: https://github.com/cloudfoundry/cf-java-client/commits?author=twoseat
[6]: https://github.com/cloudfoundry/cf-java-client/commits?author=glyn
[7]: https://github.com/cloudfoundry/cf-java-client/commits?author=LokeshN
[8]: https://github.com/cloudfoundry/cf-java-client/commits?author=o-orand
[9]: https://github.com/cloudfoundry/cf-java-client/commits?author=s-bortolussi
[10]: https://github.com/cloudfoundry/cf-java-client/commits?author=smaldini
[11]: https://github.com/cloudfoundry/cf-java-client/commits?author=benpatt
[12]: https://github.com/cloudfoundry/cf-java-client/commits?author=mheath
[13]: https://github.com/cloudfoundry/cf-java-client/commits?author=ramnivas
[14]: https://github.com/cloudfoundry/cf-java-client/commits?author=WGautier
[15]: https://projectreactor.io
[16]: https://screenhero.com
[17]: https://github.com/spring-cloud/spring-cloud-dataflow
[18]: https://github.com/spring-cloud/spring-cloud-spinnaker
[19]: https://github.com/cloudfoundry/cf-java-client/commit/dc8f8d5c8370de223992a0e4a024d5cff082a682
[20]: http://www.reactive-streams.org
[21]: https://github.com/cloudfoundry/cf-java-client/releases/tag/v2.0.0.M1
[22]: https://github.com/cloudfoundry/cf-java-client/tree/v2.0.0.RELEASE
[23]: http://cloudfoundry.github.io/cf-java-client/api/latest-release/cloudfoundry-client
[24]: http://cloudfoundry.github.io/cf-java-client/api/latest-release/cloudfoundry-client-reactor
[25]: http://cloudfoundry.github.io/cf-java-client/api/latest-release/cloudfoundry-operations
[26]: http://cloudfoundry.github.io/cf-java-client/api/latest-release/cloudfoundry-util
[27]: http://apidocs.cloudfoundry.org/latest-release
[28]: http://v3-apidocs.cloudfoundry.org/version/release-candidate/index.html
[29]: https://github.com/cloudfoundry/loggregator/tree/develop/src/trafficcontroller#endpoints
[30]: http://docs.cloudfoundry.com/uaa
[31]: https://github.com/cloudfoundry/cf-java-client/issues
[32]: https://github.com/cloudfoundry/cf-java-client/pulls
[33]: https://cloudfoundry.slack.com/messages/java-client/


Re: DNS caching and forwarding in Cloud Foundry with Consul Agents

Amit Kumar Gupta
 

Hi Hector,

If you are able to configure consul-release to (a) not rewrite
/etc/resolv.conf and (b) not listen on port 53, then I assume you would run
your own thing like bind or dnsmasq there, and only forward specific
queries to the consul agent. In this case, why would you need to specify
which recursors consul hits, given that you would probably never send it
queries outside consul's domain anyways, thus it would never try any
recursing?

Best,
Amit

On Mon, Aug 22, 2016 at 5:21 AM, Hector Rivas Gandara <
hector.rivas.gandara(a)digital.cabinet-office.gov.uk> wrote:

This all sounds great. Sounds like you just want options to (1)
configure
which port Consul DNS binds to, and (2) toggle whether or not it makes
changes to /etc/resolv.conf, at minimum.
Yeah, that would be the minimum required.

Then, I would add:

(3) override the recursors to be used by consul.

With those 3x options, consul can be configured to support any
scenario without having to fork any release.

Is this something you'd want to configure Consul to do, or
would you configure your upstream component like dnsmasq or bind to only
hit
the Consul DNS server port when it receives a query for that search
domain?

Yes, that should be the pattern.

You also mentioned you don't like Consul's recursor behaviour, and want
to
"turn it off". Independently, other people have asked for Consul to be
able
to do more recursing, by being able to provide it a list of IPs to
recurse
to *in addition* to the ones it pulls out of /etc/resolv.conf. So I see
a
few use cases people might have:

(a) make Consul not do any recursing at all, and let the logic live
upstream
(in something like dnsmasq or bind) to try a different server when Consul
isn't able to resolve an address
(b) doesn't matter at all what Consul recursing does, because you want to
configure something like dnsmasq to only send queries to Consul when it
matches a certain domain, so no matter how Consul is configured to
recurse,
in practice this would never happen
(c) doesn't matter at all what Consul recursing does, because you want to
configure Consul itself to only handle queries which match a certain
domain,
so no matter how Consul is configured to recurse, in practice this would
never happen
(d) be able to override the set of recursors Consul uses (instead of
what it
reads out of /etc/resolv.conf), so that you could set it to an empty
array
or whatever you like -- this is one way to generalize use case (a)
(e) be able to add a set of recursors to the ones Consul reads out of
/etc/resolv.conf, because the stuff in /etc/resolv.conf is determined by
the
BOSH Director, and not necessarily something you want to mess with, but
may
be something you want to add to, to get over limitations like "MAXNS 3".
See: https://www.pivotaltracker.com/story/show/121335409

If consul-release makes it possible to change the binding port and
toggle it
not to modify /etc/resolv.conf, then would you say your use cases falls
into
bucket (b)? If so, that's good because it requires fewer additional
changes
and also supports consul-release being able to support use case (e) as
well
at some point, without conflict.
Yes, I think that would be the case.

IMO I think it is enough to allow the cases b and d. That can be done
by implementing the features 1, 2 and 3 mention before.


(e) be able to add a set of recursors to the ones Consul reads out of
/etc/resolv.conf, because the stuff in /etc/resolv.conf is determined by
the
BOSH Director, and not necessarily something you want to mess with, but
may
be something you want to add to, to get over limitations like "MAXNS 3".
See: https://www.pivotaltracker.com/story/show/121335409
Rare case and suboptimal.

Be aware that in the current consul implementation, each recursor will
be queried in round robin, with a hardcoded timeout of 2 seconds. So,
if you have 4x recursors, you query might tike 8s to complete! and
given that the default timeout in linux is 5 seconds with 2x retries,
you will have to override that as well.

As I commented in a previous mail, the consul recursor logic is not
smart at all, compared with other agents like dnsmasq, pdnsd or bind.
If somebody wants to customise and improve the resolution logic, they
should look at those agents, not at consul.

Because that, I don't think it makes sense to put an option of "add
new recursors to the list". Instead limit it to optionally specify the
full list of DNS to be configured as recursors (and empty to disable).
"Add recursors" is a rare case and I think it is OK to have to a
little bit of duplication of defining the DNS nameservers twice.



--
Regards
Hector Rivas | GDS / Multi-Cloud PaaS


Re: DNS caching and forwarding in Cloud Foundry with Consul Agents

Hector Rivas Gandara
 

This all sounds great. Sounds like you just want options to (1) configure
which port Consul DNS binds to, and (2) toggle whether or not it makes
changes to /etc/resolv.conf, at minimum.
Yeah, that would be the minimum required.

Then, I would add:

(3) override the recursors to be used by consul.

With those 3x options, consul can be configured to support any
scenario without having to fork any release.

Is this something you'd want to configure Consul to do, or
would you configure your upstream component like dnsmasq or bind to only hit
the Consul DNS server port when it receives a query for that search domain?
Yes, that should be the pattern.

You also mentioned you don't like Consul's recursor behaviour, and want to
"turn it off". Independently, other people have asked for Consul to be able
to do more recursing, by being able to provide it a list of IPs to recurse
to *in addition* to the ones it pulls out of /etc/resolv.conf. So I see a
few use cases people might have:

(a) make Consul not do any recursing at all, and let the logic live upstream
(in something like dnsmasq or bind) to try a different server when Consul
isn't able to resolve an address
(b) doesn't matter at all what Consul recursing does, because you want to
configure something like dnsmasq to only send queries to Consul when it
matches a certain domain, so no matter how Consul is configured to recurse,
in practice this would never happen
(c) doesn't matter at all what Consul recursing does, because you want to
configure Consul itself to only handle queries which match a certain domain,
so no matter how Consul is configured to recurse, in practice this would
never happen
(d) be able to override the set of recursors Consul uses (instead of what it
reads out of /etc/resolv.conf), so that you could set it to an empty array
or whatever you like -- this is one way to generalize use case (a)
(e) be able to add a set of recursors to the ones Consul reads out of
/etc/resolv.conf, because the stuff in /etc/resolv.conf is determined by the
BOSH Director, and not necessarily something you want to mess with, but may
be something you want to add to, to get over limitations like "MAXNS 3".
See: https://www.pivotaltracker.com/story/show/121335409

If consul-release makes it possible to change the binding port and toggle it
not to modify /etc/resolv.conf, then would you say your use cases falls into
bucket (b)? If so, that's good because it requires fewer additional changes
and also supports consul-release being able to support use case (e) as well
at some point, without conflict.
Yes, I think that would be the case.

IMO I think it is enough to allow the cases b and d. That can be done
by implementing the features 1, 2 and 3 mention before.


(e) be able to add a set of recursors to the ones Consul reads out of
/etc/resolv.conf, because the stuff in /etc/resolv.conf is determined by the
BOSH Director, and not necessarily something you want to mess with, but may
be something you want to add to, to get over limitations like "MAXNS 3".
See: https://www.pivotaltracker.com/story/show/121335409
Rare case and suboptimal.

Be aware that in the current consul implementation, each recursor will
be queried in round robin, with a hardcoded timeout of 2 seconds. So,
if you have 4x recursors, you query might tike 8s to complete! and
given that the default timeout in linux is 5 seconds with 2x retries,
you will have to override that as well.

As I commented in a previous mail, the consul recursor logic is not
smart at all, compared with other agents like dnsmasq, pdnsd or bind.
If somebody wants to customise and improve the resolution logic, they
should look at those agents, not at consul.

Because that, I don't think it makes sense to put an option of "add
new recursors to the list". Instead limit it to optionally specify the
full list of DNS to be configured as recursors (and empty to disable).
"Add recursors" is a rare case and I think it is OK to have to a
little bit of duplication of defining the DNS nameservers twice.



--
Regards
Hector Rivas | GDS / Multi-Cloud PaaS


CF Routing Use Cases

Moiz Arif <moizarif2002@...>
 

Hi All,
I have put together a slide deck explaining a few routing use cases based on my understanding of the traffic flows in Cloud Foundry.
Please feel free to point out any missing/erroneous flows in the deck. So, that we can make this deck a good learning experience for users new to CF. Your feedback will be greatly appreciated :-)
ThanksMoiz Arif


Re: Runtime PMC - Loggregator Project Lead

Gwenn Etourneau
 

It was a pleasure to work with Jim as well as colleague and as OSS
contributor (firehose-to-syslog).
Jim always help us to bring the best to the loggregator.

On Fri, Aug 19, 2016 at 11:44 PM, Dieu Cao <dcao(a)pivotal.io> wrote:

Hello all,

Jim Campbell, the Project Lead for the Loggregator team within the Runtime
PMC, last day at Pivotal was this week. We wish him well in his future
endeavors and thank him for his time with us.

The Loggregator team, primarily located in Denver, Colorado, now has an
opening for its project lead. Please send nominations to me. Project
leads must be nominated by a Cloud Foundry Foundation member.

In the interim, Pivotal would like to nominate Allen Duet, a PM in Denver,
to fill in as the Project Lead for Loggregator until a new Project Lead is
selected. If there are any questions objections to this temporary
appointment, please send that to me by end of day August 25, 2016.

If you have any questions about the role/process, please let me know.
These are described in the CFF governance documents. [1]

-Dieu Cao

Runtime PMC Lead


Re: UAA release manifest changes - Action required

Filip Hanik
 

We're planning on the next cf-release bump. At this time, both formats are
supported, to allows the change to happen.

On Sat, Aug 20, 2016 at 4:05 AM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:

At what version does this change occur?

On Fri, Aug 19, 2016 at 4:41 PM, Madhura Bhave <mbhave(a)pivotal.io> wrote:

Hi Everyone,

We have made changes to the way *uaa.scim.users* and *uaa.scim.groups* can
be specified in the deployment manifest.

The *previous* format for uaa.scim.users was:

- marissa|koala|marissa(a)test.org|Marissa|Bloggs|scim.write,sci
m.read,openid

The *new* format is:
```
- name: marissa
password: koala
email: marissa(a)test.org
firstName: Marissa
lastName: Bloggs
groups:
- scim.write
- scim.read
- openid
```

The *previous* format for uaa.scim.groups was:

group1,group2,group3

The *new* format is:

group1: 'My test group description'
group2: 'My other test group description'
group3: 'My next group description'

Going forward, the *old* format will *not be supported. Please update
your stubs with the new format.*

Thanks,
Madhura
CF UAA Team


Re: UAA release manifest changes - Action required

Tom Sherrod <tom.sherrod@...>
 

At what version does this change occur?

On Fri, Aug 19, 2016 at 4:41 PM, Madhura Bhave <mbhave(a)pivotal.io> wrote:

Hi Everyone,

We have made changes to the way *uaa.scim.users* and *uaa.scim.groups* can
be specified in the deployment manifest.

The *previous* format for uaa.scim.users was:

- marissa|koala|marissa(a)test.org|Marissa|Bloggs|scim.write,sci
m.read,openid

The *new* format is:
```
- name: marissa
password: koala
email: marissa(a)test.org
firstName: Marissa
lastName: Bloggs
groups:
- scim.write
- scim.read
- openid
```

The *previous* format for uaa.scim.groups was:

group1,group2,group3

The *new* format is:

group1: 'My test group description'
group2: 'My other test group description'
group3: 'My next group description'

Going forward, the *old* format will *not be supported. Please update
your stubs with the new format.*

Thanks,
Madhura
CF UAA Team


UAA release manifest changes - Action required

Madhura Bhave
 

Hi Everyone,

We have made changes to the way *uaa.scim.users* and *uaa.scim.groups* can
be specified in the deployment manifest.

The *previous* format for uaa.scim.users was:

- marissa|koala|marissa(a)test.org|Marissa|Bloggs|scim.write,scim.read,openid

The *new* format is:
```
- name: marissa
password: koala
email: marissa(a)test.org
firstName: Marissa
lastName: Bloggs
groups:
- scim.write
- scim.read
- openid
```

The *previous* format for uaa.scim.groups was:

group1,group2,group3

The *new* format is:

group1: 'My test group description'
group2: 'My other test group description'
group3: 'My next group description'

Going forward, the *old* format will *not be supported. Please update your
stubs with the new format.*

Thanks,
Madhura
CF UAA Team


Runtime PMC - Loggregator Project Lead

Dieu Cao <dcao@...>
 

Hello all,

Jim Campbell, the Project Lead for the Loggregator team within the Runtime
PMC, last day at Pivotal was this week. We wish him well in his future
endeavors and thank him for his time with us.

The Loggregator team, primarily located in Denver, Colorado, now has an
opening for its project lead. Please send nominations to me. Project
leads must be nominated by a Cloud Foundry Foundation member.

In the interim, Pivotal would like to nominate Allen Duet, a PM in Denver,
to fill in as the Project Lead for Loggregator until a new Project Lead is
selected. If there are any questions objections to this temporary
appointment, please send that to me by end of day August 25, 2016.

If you have any questions about the role/process, please let me know.
These are described in the CFF governance documents. [1]

-Dieu Cao

Runtime PMC Lead

3841 - 3860 of 9426