Date   

Re: How to minimize VMs used in cf/diego

Eric Malm <emalm@...>
 

Hi, Stanley,

If you're looking to reduce the Diego VM footprint in a dev deployment, you
could consider using the 'colocated_vN' (N=1,2,3) jobs in the manifests
generated by the spiff-based scripts in diego-release. They pack all the
jobs in diego-release (and garden and etcd) onto a single VM, so you could
deploy with 1 instance of colocated_z1 and 0 instances of all other jobs.
If you need more container capacity in that deployment, you can then
increase the instance count of the cell jobs from 0 (you will have one cell
already via the existing colocated job instance).

Also, as a warning, the Diego team doesn't regularly test the colocated-VM
deployment configuration, so it's possible that it may accidentally break
in the future (or even that it's broken now!). So I wouldn't recommend its
use in a production deployment.

I can't speak to the current state of colocatability of the jobs in
cf-release, or to the status of the community cf-boshworkspace project.

Best,
Eric, CF Runtime Diego PM

On Wed, Mar 16, 2016 at 8:33 PM, Stanley Shen <meteorping(a)gmail.com> wrote:

Hello, all

By default, there are about 20 VMs will be used in cf/diego installation.
I am trying to minimize the VM size.

I tried to follow
https://github.com/cloudfoundry-community/cf-boshworkspace but got some
problem.
I create an issue for it but got no response, is this still maintained?

If there are any other documentation about it, or any experience on it?



Reg the upgrade from cf-205 to cf-231

Nithiyasri Gnanasekaran -X (ngnanase - TECH MAHINDRA LIM@Cisco) <ngnanase at cisco.com...>
 

Hi Amit

I was able to do an upgrade of deployment from cf-205 to cf-231 multiple times.
But now, I am getting timeout errors while compiling buildpacks once in rootfs or in buildpack_java or in nginx..
Could you pls help us resolve this error..

Following is the one:

Done compiling packages > buildpack_java/d65c2d20fc067c9995c18d195e0ff117ea9202c0 (00:24:23)
Done compiling packages > rtr/2d7de4f6fc25938c21c5be87174f95583feb14b5 (00:38:22)
Failed compiling packages > rootfs_cflinuxfs2/ac0ba35f1106af81cb735eb56360c6cc924af83a: Timed out waiting for Server `3c04cecc-1308-4da f-8dc5-b9c1924004c3' to be active (00:41:23)
Failed compiling packages > nginx/bf3af6163e13887aacd230bbbc5eff90213ac6af: Timed out waiting for Server `fbbb8647-1415-46d9-92de-d030f 4164b6b' to be active (00:43:31)



Failed compiling packages > cli/b61b75716312df9f4f7c81a17f3a7de456efce71: Timed out waiting for Server `223ec673-b66d-4362-8e27-0859805 2ebe5' to be active (00:44:49)

Error 100: Timed out waiting for Server `3c04cecc-1308-4daf-8dc5-b9c1924004c3' to be active


Regards
Nithiyasri


Re: Using swift as a blobstore in cloud foundry with keystone v3

Marco Voelz
 

Dear Nicholas,

if desired, we can also do a PR for allowing the CC to connect to Swift using Keystone v3. A couple of months ago we did the same thing in the OpenStack CPI. We also have some test envs available where we can validate the change. What do you think?

Warm regards
Marco

On 16/03/16 18:13, "Nicholas Calugar" <ncalugar(a)pivotal.io<mailto:ncalugar(a)pivotal.io>> wrote:

Hi Muhammad,

Unfortunately, we don't have an environment using keystone v3. We are passing the configuration to fog as-is. There are several fixes that have been made in later releases of fog, for example:

https://github.com/fog/fog/pull/3806

I'll get a story prioritized to upgrade fog to v1.37.0

Thanks,

Nick

On Sun, Mar 13, 2016 at 11:58 PM Altaf, Muhammad <Muhammada(a)fast.au.fujitsu.com<mailto:Muhammada(a)fast.au.fujitsu.com>> wrote:
Hi All,
I am trying to configure cloud foundry to use swift on OpenStack. I have followed the instructions at https://docs.cloudfoundry.org/deploying/openstack/using_swift_blobstore.html
When used keystone v2, I am able to start my apps on DEA which is good. However when using keystone V3, I am not able to start my apps. The error I am getting is:

“FAILED
Server error, status code: 400, error code: 170001, message: Staging error: failed to stage application:
Error downloading: HTTP status: 401”

Tried to debug by adding some ‘puts’ statements in openstack/core.rb file and it looks like tokens are being generated successfully so there is no problem with the authentication. The generated response to auth request shows that the user has “ResellerAdmin” role as well.

When I look into runner_z1/0 /var/vcap/data/dea_next/tmp/ app-package-download.tgz2016*, I find error saying: “401 Unauthorized: Temp URL invalid xxxxx”

/var/vcap/sys/log/dea_next/dea_next.log shows some download URLs, and if I curl those URLs, I get exact same error message. Below are the fog_connection settings in cloud foundry manifest:

fog_connection: &fog_connection
provider: 'OpenStack'
openstack_username: 'cf-admin2'
openstack_tenant: 'cf2'
openstack_project_name: 'cf2'
openstack_api_key: 'passw0rd'
openstack_auth_url: 'http://<OPENSTACK_IP>:5000/v3/auth/tokens'
openstack_domain_name: 'cf_domain'
openstack_user_domain_name: 'cf_domain'
openstack_temp_url_key: 'b3968d0207b54ece87cccc06515a89d4'


Account has a valid temp_url_key configured. Please see below:
curl -v -X GET http://SWIFT_IP:SWIFT_PORT/v2/Auth_b34a51e551ec4796a461168c886c734f -H "X-Auth-Token: TOKEN"
* Hostname was NOT found in DNS cache
* Trying SWIFT_IP...
* Connected to SWIFT_IP (SWIFT_IP) port SWIFT_PORT (#0)
GET /v2/Auth_b34a51e551ec4796a461168c886c734f HTTP/1.1
User-Agent: curl/7.35.0
Host: SWIFT_IP:SWIFT_PORT
Accept: */*
X-Auth-Token: TOKEN
< HTTP/1.1 204 No Content
< Content-Length: 0
< X-Account-Object-Count: 0
< X-Timestamp: 1457918518.21777
< X-Account-Meta-Temp-Url-Key: b3968d0207b54ece87cccc06515a89d4
< X-Account-Bytes-Used: 0
< X-Account-Container-Count: 0
< Content-Type: text/plain; charset=utf-8
< Accept-Ranges: bytes
< X-Trans-Id: txfc362c27bdda4355a942a-0056e65d93
< Date: Mon, 14 Mar 2016 06:43:31 GMT
<
* Connection #0 to host SWIFT_IP left intact

Also, I can see that the containers are created on swift, so obviously it is able to authenticate.
$ openstack container list
+---------------+
| Name |
+---------------+
| cc-buildpacks |
| cc-droplets |
| cc-packages |
| cc-resources |
+---------------+

I would appreciate if someone can help me fixing this issue.

Regards,
Muhammad Altaf
Software Development Engineer

Fujitsu Australia Software Technology Pty Ltd
14 Rodborough Road, Frenchs Forest NSW 2086, Australia
T +61 2 9452 9067 F +61 2 9975 2899
Muhammada(a)fast.au.fujitsu.com<mailto:Muhammada(a)fast.au.fujitsu.com>
fastware.com.au<http://fastware.com.au>

[image001.jpg]
[image002.jpg]

Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.


Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.


If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe(a)fast.au.fujitsu.com<mailto:unsubscribe(a)fast.au.fujitsu.com>


Re: cloud_connector_ng not starting on cf 231

Christian Schwerdtfeger
 

after a lot of investigation, we found out, that it was simply a "duplicate domain" problem cause we set system_domain and domain to the same value. so this is solved.


How to minimize VMs used in cf/diego

Stanley Shen <meteorping@...>
 

Hello, all

By default, there are about 20 VMs will be used in cf/diego installation.
I am trying to minimize the VM size.

I tried to follow https://github.com/cloudfoundry-community/cf-boshworkspace but got some problem.
I create an issue for it but got no response, is this still maintained?

If there are any other documentation about it, or any experience on it?


CF manifest changes required: java buildpack extracted

Amit Kumar Gupta
 

Hey cf-dev,

Completing the work from CF v231 to extract the buildpacks to a separate
release, the java buildpacks have also been extracted as of CF v232. As
before, the new release is submoduled back into cf-release and the new jobs
symlinked so that you have the option to continue using the buildpacks with
minimal changes to your manifest if you don't want to take advantage of the
extraction.

To continue with minimal changes, simply add:

- name: java-buildpack
release: cf

or

- name: java-offline-buildpack
release: cf

to the templates section of whatever job you have running the
cloud_controller_ng job.

If you're using the "spiff" based manifest generation tooling in
cf-release, and you're not already overriding the api_templates, you will
pick up this change for free when you regenerate your deployment manifest.

Best,
Amit


Re: Using swift as a blobstore in cloud foundry with keystone v3

Altaf, Muhammad
 

Thanks Nicholas,
Please let me know when you have an update on this.

Regards,
Muhammad Altaf
Software Development Engineer

Fujitsu Australia Software Technology Pty Ltd
14 Rodborough Road, Frenchs Forest NSW 2086, Australia
T +61 2 9452 9067 F +61 2 9975 2899
Muhammada(a)fast.au.fujitsu.com<mailto:Muhammada(a)fast.au.fujitsu.com>
fastware.com.au<http://fastware.com.au>

[cid:image001.jpg(a)01D18036.84A91D80]
[cid:image002.jpg(a)01D18036.84A91D80]
From: Nicholas Calugar [mailto:ncalugar(a)pivotal.io]
Sent: Thursday, 17 March 2016 4:14 AM
To: Discussions about Cloud Foundry projects and the system overall.
Subject: [cf-dev] Re: Using swift as a blobstore in cloud foundry with keystone v3

Hi Muhammad,

Unfortunately, we don't have an environment using keystone v3. We are passing the configuration to fog as-is. There are several fixes that have been made in later releases of fog, for example:

https://github.com/fog/fog/pull/3806

I'll get a story prioritized to upgrade fog to v1.37.0

Thanks,

Nick

On Sun, Mar 13, 2016 at 11:58 PM Altaf, Muhammad <Muhammada(a)fast.au.fujitsu.com<mailto:Muhammada(a)fast.au.fujitsu.com>> wrote:
Hi All,
I am trying to configure cloud foundry to use swift on OpenStack. I have followed the instructions at https://docs.cloudfoundry.org/deploying/openstack/using_swift_blobstore.html
When used keystone v2, I am able to start my apps on DEA which is good. However when using keystone V3, I am not able to start my apps. The error I am getting is:

“FAILED
Server error, status code: 400, error code: 170001, message: Staging error: failed to stage application:
Error downloading: HTTP status: 401”

Tried to debug by adding some ‘puts’ statements in openstack/core.rb file and it looks like tokens are being generated successfully so there is no problem with the authentication. The generated response to auth request shows that the user has “ResellerAdmin” role as well.

When I look into runner_z1/0 /var/vcap/data/dea_next/tmp/ app-package-download.tgz2016*, I find error saying: “401 Unauthorized: Temp URL invalid xxxxx”

/var/vcap/sys/log/dea_next/dea_next.log shows some download URLs, and if I curl those URLs, I get exact same error message. Below are the fog_connection settings in cloud foundry manifest:

fog_connection: &fog_connection
provider: 'OpenStack'
openstack_username: 'cf-admin2'
openstack_tenant: 'cf2'
openstack_project_name: 'cf2'
openstack_api_key: 'passw0rd'
openstack_auth_url: 'http://<OPENSTACK_IP>:5000/v3/auth/tokens'
openstack_domain_name: 'cf_domain'
openstack_user_domain_name: 'cf_domain'
openstack_temp_url_key: 'b3968d0207b54ece87cccc06515a89d4'


Account has a valid temp_url_key configured. Please see below:
curl -v -X GET http://SWIFT_IP:SWIFT_PORT/v2/Auth_b34a51e551ec4796a461168c886c734f -H "X-Auth-Token: TOKEN"
* Hostname was NOT found in DNS cache
* Trying SWIFT_IP...
* Connected to SWIFT_IP (SWIFT_IP) port SWIFT_PORT (#0)
GET /v2/Auth_b34a51e551ec4796a461168c886c734f HTTP/1.1
User-Agent: curl/7.35.0
Host: SWIFT_IP:SWIFT_PORT
Accept: */*
X-Auth-Token: TOKEN
< HTTP/1.1 204 No Content
< Content-Length: 0
< X-Account-Object-Count: 0
< X-Timestamp: 1457918518.21777
< X-Account-Meta-Temp-Url-Key: b3968d0207b54ece87cccc06515a89d4
< X-Account-Bytes-Used: 0
< X-Account-Container-Count: 0
< Content-Type: text/plain; charset=utf-8
< Accept-Ranges: bytes
< X-Trans-Id: txfc362c27bdda4355a942a-0056e65d93
< Date: Mon, 14 Mar 2016 06:43:31 GMT
<
* Connection #0 to host SWIFT_IP left intact

Also, I can see that the containers are created on swift, so obviously it is able to authenticate.
$ openstack container list
+---------------+
| Name |
+---------------+
| cc-buildpacks |
| cc-droplets |
| cc-packages |
| cc-resources |
+---------------+

I would appreciate if someone can help me fixing this issue.

Regards,
Muhammad Altaf
Software Development Engineer

Fujitsu Australia Software Technology Pty Ltd
14 Rodborough Road, Frenchs Forest NSW 2086, Australia
T +61 2 9452 9067 F +61 2 9975 2899
Muhammada(a)fast.au.fujitsu.com<mailto:Muhammada(a)fast.au.fujitsu.com>
fastware.com.au<http://fastware.com.au>
[image001.jpg]
[image002.jpg]
Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.


Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.


If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe(a)fast.au.fujitsu.com<mailto:unsubscribe(a)fast.au.fujitsu.com>

Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.


Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.


If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe(a)fast.au.fujitsu.com


Re: Announcing cf-mysql-release v26, now with a slimmer VM footprint!

Marco Nicosia
 

On Thu, Mar 10, 2016 at 6:08 AM, Benjamin Gandon <benjamin(a)gandon.org>
wrote:

Thanks for the answer Marco! (I realize I'm much influenced by the
Percona vision, because they are active here in Paris dev communities.)

I finally upgraded to v26 and well.. Congratulations for the deep review
of manifests, but.. What a tremendous change!
Yeah, I acknowledge that it's a significant impact, and I regret that we
weren't able to come up with an easier way to do it. However, in order to
make a material difference in how difficult it is to generate manifests, it
was necessary to fundamentally re-factor the properties.


My feedback: starting from a mere mistake forgetting the "http://" prefix
in the "api_url" prop, I arrived to a half-finished bosh deploy :(
By the time I realized the issue was not about the broker database
migration (which was failing and prevented the az1 broker from booting) I
already had wiped the cluster out.
This is very concerning. Typically, when a bosh deploy fails to complete,
you should be able to fix the manifest issue and roll forward.

I am worried about the broker database migration, it should be idempotent.
If you have more information that would allow us to better understand,
please open a GitHub issue so that we can look into it further.


Fortunately it was not in production! But the manifest change is so big
that such mistakes are easy to do.
How do you generate manifests?

We have made assumptions that Operators are maintaining and updating Spiff
stubs, and generating an entirely new manifest with each release. If this
is not the case, please let us know how you maintain your manifests,
release over release.

Anyway I wish we had some basic syntax checks on Bosh props that would
have warned me like: “Hey you need an 'http://' prefix here!”
We don't have validator functionality that we can offer right now. However,
we provide scripts to generate manifests, which, among other things,
provide the api_url automatically
<https://github.com/cloudfoundry/cf-mysql-release/blob/v26/manifest-generation/misc-templates/config-from-cf-internal.yml#L12>
(hence
my question above).

Btw, white labeling is not finished: there are still a bunch of Pivotal
references out there. Didn't you know that the "p" in "p-mysql" stands for
"Pivotal"? ;)
This is an artifact of the fact that, originally, we had hoped that there
might be many providers of MySQL in the marketplace. So although this is
the Cloud Foundry Foundation's software, we use the "p-" to distinguish it
as software that was primarily contributed by Pivotal. If others wanted to
publish MySQL services, they would show up alongside p-mysql when a user
runs `cf marketplace.`

For what it's worth, you can change the manifest property, `
properties.cf-mysql.broker.name` to whatever name you'd like to show in the
marketplace.

I wrote a story <https://www.pivotaltracker.com/story/show/115805005> to
get this and the api_url
<https://www.pivotaltracker.com/story/show/115819685>, documented in the
spec files.

I might push some PR within the next days or so about that.

/Benjamin
--
Marco Nicosia
Product Manager
Pivotal Software, Inc.



Le 2 mars 2016 à 21:16, Marco Nicosia <mnicosia(a)pivotal.io> a écrit :

Hi Benjamin,

Sorry for the delayed response!

There are technically no trade-offs when using an Arbitrator, with the
sole exception that you're sacrificing an extra level of data redundancy by
keeping only two copies of the data, not three.

For that reason, a three node cluster is still a "standard" deployment
option, you just use the no-arbitrator
<https://github.com/cloudfoundry/cf-mysql-release/tree/v26/manifest-generation/examples/no-arbitrator>
example during manifest generation.

Galera doesn't do GTID re-sync in the normal way that MySQL does. GTID has
a slightly different context. There's a blog
<https://www.percona.com/blog/2015/02/13/percona-xtradb-cluster-5-6-a-tale-of-2-mysql-gtids/>
that describes some of the differences if you want to dive in. In the case
of whole-cluster failure, we describe how to bootstrap
<https://github.com/cloudfoundry/cf-mysql-release/blob/v26/docs/bootstrapping.md>
the cluster in the documentation.

Finally, 10.1 is very definitely something we're excited to begin to
support. It's something we want to do in a way that will allow users to
migrate as they feel comfortable. So, we have some challenges to figure out
how to make it an easy option while still allowing conservative users 10.0
as an option.

--
Marco Nicosia
Product Manager
Pivotal Software, Inc.


On Fri, Feb 26, 2016 at 2:34 AM, Benjamin Gandon <benjamin(a)gandon.org>
wrote:

Hi,

Congratulations for this v26 release!

Do you have any documentation for the benefits and trade offs
introduced by this new arbitrator, compared to the standard 3-nodes setup?
What happens to the "quorum" principle in such arbitrator setup?

Are there any consequences or benefits in terms of managing possible sync
failures and manual GTID re-sync?

And a last question : do you plan upgrading to MariaDB 10.1.x in future
releases?

/Benjamin

Le 25 févr. 2016 à 17:53, Mike Youngstrom <youngm(a)gmail.com> a écrit :

In our case we'll use the arbitrator because we only have 2 AZs in some
datacenters. The Arbitrator allows us to place the 3rd member of the
cluster in another datacenter minimal performance impact. Nice feature!

Mike

On Thu, Feb 25, 2016 at 9:49 AM, Duncan Winn <dwinn(a)pivotal.io> wrote:

+1.. The arbitrator is a fantastic feature. Great job MySQL team.

On Thu, 25 Feb 2016 at 00:39, James Bayer <jbayer(a)pivotal.io> wrote:

congrats mysql team! the arbitrator is a nice touch to save resources
yet still result in high availability even when losing an entire AZ.

On Wed, Feb 24, 2016 at 10:43 AM, Marco Nicosia <mnicosia(a)pivotal.io>
wrote:

Hello Cloud Foundry folks,

For those of you who are tired of deploying web apps without easy
integration with data services, cf-mysql is a great place to start. With
this release, I'm happy to tell you that it takes even less commitment than
ever to give cf-mysql a spin!

The theme for this release is the First Rule in Government Spending
<https://www.youtube.com/watch?v=Et4sMJP9FmM>. Wanna take a ride?

A single MySQL node is a Single Point of Failure. With the
introduction of the Arbitrator, we allow you to buy two at twice the price.
Previously, we made you buy *three* for the same sense of security.
Upgrading to cf-mysql v26 will *save you money*, with no sacrifice in
performance!


If you like what we're doing with cf-mysql, click the thumbs up: 👍
<http://clickpoint.cfapps.io/click/mysql-v26?up>
If we're messing up your game, click the thumbs down: 👎
<http://clickpoint.cfapps.io/click/mysql-v26?down>
And if this just isn't your thing, at least give me a fist bump! 👊
<http://clickpoint.cfapps.io/click/mysql-v26?meh>

*Highlights*

- We've updated to *MariaDB 10.0.23*, enabled some important
behind-the-scenes features, and fixed several important bugs.
- For those of you who require audit access to all data, we've given
you the option to enable a *Read Only admin user*.
- We've introduced a new HA deployment option, *2+1: *two MySQL nodes
and an *Arbitrator*.
- And we've made significant updates to the *generate-deployment-manifest
script* and stubs.


You'll have to update your stubs to use the new release, but we hope
with the new Examples and Arbitrator feature, it'll be worth your time to
upgrade. Make sure to read the release notes for important information
about the upgrade process.

As always, for full disclosure, and links beyond that, please check
out the Release Notes
<https://github.com/cloudfoundry/cf-mysql-release/releases/tag/v26>.

*Introducing the Arbitrator*

With cf-mysql v26, we've replaced one of the MySQL nodes with a
lightweight Arbitrator node. Previously, the minimal HA configuration
required three full-size MySQL nodes.

For cf-mysql administrators who are careful with their infrastructure
resources, the Arbitrator feature is a new deployment topology that uses a
smaller VM footprint while maintaining high availability guarantees. Unlike
the old three node topology, the Arbitrator decreases spend with no impact
on performance.

Thanks, and make sure to give me your feedback to influence what we do
for future releases!

--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io


--
Thank you,

James Bayer
--
Duncan Winn
Cloud Foundry PCF Services


Spiff reloaded...

Krueger, Uwe <uwe.krueger@...>
 

Is there still somebody out there using spiff?

Then you might be interested in some good news...

End of last year I've started to take a look into the source code.
After providing some major fixes for the official spiff release
I began to introduce new basic functionality that is very useful
for generating deployment manifests, in the context of multiple
cooperating deployments.

Unfortunately the development of the original spiff version is paused,
so, besides the fixes and some minor improvements, no more feature pull
requests were accepted.

Therefore I decided to offer the new functionality in a forked repository.
Here the first released version of spiff++ is out now. As usual with binaries
for linux and darwin.

In contrast to usual preprocessing based solutions for generating
yaml based deployment manifests, spiff completely works in the domain of
the yaml document. This offers the possibility to directly work with the
structured data in the yaml document without the need of thinking in
two description domains.

Spiff++ now significantly improves the possibilities to describe complex
structures in yaml documents based on dynaml expressions embedded in
the yaml documents with full access to the complete document.

- fully compatible with spiff
- as long as development of spiff is paused all fixes will be incorporated
- improved error reporting
- complete set of basic integer arithmetic
- arithmetic and built-in functions based on IPv4 addresses and CIDRs
- logical operators and conditional expressions
- rich set of new built-in functions
- joining of lists and maps
- extended multi-document merging features
- extendable by user defined functions as part of the yaml document by
introducing lambda expressions
- mapping support based on yaml nodes
- access to any foreign data source by consuming outputs of OS commands
- access to the file system
- dynamic in-document template processing

If you want to know more, please have a look at the documentation at

https://github.com/mandelsoft/spiff/blob/master/README.md

there you also find some examples how spiff++ can simplify your manifest
generation.

Happy spiffing
Uwe


[ANN] nodejs-buildpack and OpenSSL vendoring

JT Archie <jarchie@...>
 

Since November 2015, the [nodejs-buildpack](https://github.com/cloudfoundry/nodejs-buildpack) has been packaging binaries of Node.js® with OpenSSL that are statically linked. With [community approval](https://github.com/cloudfoundry/nodejs-buildpack/issues/32), it was decided to support Node.js® 4.x and greater, which relied on the Node.js® release cycle to provide OpenSSL updates.

The buildpack's team [binary-builder](https://github.com/cloudfoundry/binary-builder) was updated to [enable the static openssl compilation](https://github.com/cloudfoundry/binary-builder/commit/834759affa4d7e42294a54b49bac6f1cf81b798a). All versions of Node.js® compiled since have been statically linked with OpenSSL, which include versions of Node.js® greater than and equal to 0.12.10 and 0.10.42.

Kind Regards,

JT and Kelly (Cloud Foundry buildpack team members)


Re: Using swift as a blobstore in cloud foundry with keystone v3

Nicholas Calugar
 

Hi Muhammad,

Unfortunately, we don't have an environment using keystone v3. We are
passing the configuration to fog as-is. There are several fixes that have
been made in later releases of fog, for example:

https://github.com/fog/fog/pull/3806

I'll get a story prioritized to upgrade fog to v1.37.0

Thanks,

Nick

On Sun, Mar 13, 2016 at 11:58 PM Altaf, Muhammad <
Muhammada(a)fast.au.fujitsu.com> wrote:

Hi All,

I am trying to configure cloud foundry to use swift on OpenStack. I have
followed the instructions at
https://docs.cloudfoundry.org/deploying/openstack/using_swift_blobstore.html

When used keystone v2, I am able to start my apps on DEA which is good.
However when using keystone V3, I am not able to start my apps. The error I
am getting is:



“FAILED

Server error, status code: 400, error code: 170001, message: Staging
error: failed to stage application:

Error downloading: HTTP status: 401”



Tried to debug by adding some ‘puts’ statements in openstack/core.rb file
and it looks like tokens are being generated successfully so there is no
problem with the authentication. The generated response to auth request
shows that the user has “ResellerAdmin” role as well.



When I look into runner_z1/0 /var/vcap/data/dea_next/tmp/
app-package-download.tgz2016*, I find error saying: “401 Unauthorized:
Temp URL invalid xxxxx”



/var/vcap/sys/log/dea_next/dea_next.log shows some download URLs, and if I
curl those URLs, I get exact same error message. Below are the
fog_connection settings in cloud foundry manifest:



fog_connection: &fog_connection

provider: 'OpenStack'

openstack_username: 'cf-admin2'

openstack_tenant: 'cf2'

openstack_project_name: 'cf2'

openstack_api_key: 'passw0rd'

openstack_auth_url: 'http://<OPENSTACK_IP>:5000/v3/auth/tokens'

openstack_domain_name: 'cf_domain'

openstack_user_domain_name: 'cf_domain'

openstack_temp_url_key: 'b3968d0207b54ece87cccc06515a89d4'





Account has a valid temp_url_key configured. Please see below:

curl -v -X GET http://SWIFT_IP:SWIFT_PORT/v2/Auth_b34a51e551ec4796a461168c886c734f
-H "X-Auth-Token: TOKEN"

* Hostname was NOT found in DNS cache

* Trying SWIFT_IP...

* Connected to SWIFT_IP (SWIFT_IP) port SWIFT_PORT (#0)

GET /v2/Auth_b34a51e551ec4796a461168c886c734f HTTP/1.1
User-Agent: curl/7.35.0
Host: SWIFT_IP:SWIFT_PORT
Accept: */*
X-Auth-Token: TOKEN
< HTTP/1.1 204 No Content

< Content-Length: 0

< X-Account-Object-Count: 0

< X-Timestamp: 1457918518.21777

< X-Account-Meta-Temp-Url-Key: *b3968d0207b54ece87cccc06515a89d4*

< X-Account-Bytes-Used: 0

< X-Account-Container-Count: 0

< Content-Type: text/plain; charset=utf-8

< Accept-Ranges: bytes

< X-Trans-Id: txfc362c27bdda4355a942a-0056e65d93

< Date: Mon, 14 Mar 2016 06:43:31 GMT

<

* Connection #0 to host SWIFT_IP left intact



Also, I can see that the containers are created on swift, so obviously it
is able to authenticate.

$ openstack container list

+---------------+

| Name |

+---------------+

| cc-buildpacks |

| cc-droplets |

| cc-packages |

| cc-resources |

+---------------+



I would appreciate if someone can help me fixing this issue.



Regards,




*Muhammad Altaf Software Development Engineer Fujitsu Australia Software
Technology Pty Ltd*
14 Rodborough Road, Frenchs Forest NSW 2086, Australia
*T* +61 2 9452 9067 *F* +61 2 9975 2899
Muhammada(a)fast.au.fujitsu.com
fastware.com.au

[image: image001.jpg]
[image: image002.jpg]

Disclaimer

The information in this e-mail is confidential and may contain content
that is subject to copyright and/or is commercial-in-confidence and is
intended only for the use of the above named addressee. If you are not the
intended recipient, you are hereby notified that dissemination, copying or
use of the information is strictly prohibited. If you have received this
e-mail in error, please telephone Fujitsu Australia Software Technology Pty
Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the
document and all copies thereof.

Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly
transmit a virus within an email communication, it is the receiver’s
responsibility to scan all communication and any files attached for
computer viruses and other defects. Fujitsu Australia Software Technology
Pty Ltd does not accept liability for any loss or damage (whether direct,
indirect, consequential or economic) however caused, and whether by
negligence or otherwise, which may result directly or indirectly from this
communication or any files attached.

If you do not wish to receive commercial and/or marketing email messages
from Fujitsu Australia Software Technology Pty Ltd, please email
unsubscribe(a)fast.au.fujitsu.com


Re: cf push "stack: unknown"

Sunil Babu <placid.babu@...>
 

Hi Dan

Check with the bosh cli options rgd the vms and status check.

On Wed, Mar 16, 2016 at 9:35 PM, Eric Malm <emalm(a)pivotal.io> wrote:

Hi, Dan,

It may be a regression in the CF CLI: in my check against one of the Diego
team environments, that stack field is consistently absent after the
initial push with CLI 6.16.1+924508c-2016-02-26, but
consistently present with 6.14.1+dc6adf6-2015-12-22 (the other version I
had at hand on my system).

Thanks,
Eric

On Wed, Mar 16, 2016 at 8:44 AM, Edward Mikuszewski <
emikuszewski(a)pivotal.io> wrote:

Great question, Dan. I'm curious to this as well - I just assumed it took
some time to report back the status.

On Wed, Mar 16, 2016 at 10:28 AM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

Quick question, when I push an app to CF I get this output at the bottom
after the app starts.

```
requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: sinatra-test-dan.cfapps.io
last uploaded: Wed Mar 16 14:54:50 UTC 2016
stack: unknown
buildpack: https://github.com/cloudfoundry/ruby-buildpack#v1.6.5

state since cpu memory disk
details
#0 running 2016-03-16 10:55:12 AM 0.1% 21M of 256M 68.3M of 1G
```

Anyone know why the stack always reports "unknown"? If I run `cf app`
right after, it will report the stack properly. It only shows up this way
on the initial push.

Versions:
- cf version 6.16.1+924508c-2016-02-26
- cf v231
- diego 0.1454.0
- garden 0.333.0

Thanks,

Dan


--
Ed Mikuszewski *| * <http://www.pivotal.io> *|* Mobile: 646-628-2872
*| *Email: emikuszewski(a)pivotal.io


Re: cf push "stack: unknown"

Eric Malm <emalm@...>
 

Hi, Dan,

It may be a regression in the CF CLI: in my check against one of the Diego
team environments, that stack field is consistently absent after the
initial push with CLI 6.16.1+924508c-2016-02-26, but
consistently present with 6.14.1+dc6adf6-2015-12-22 (the other version I
had at hand on my system).

Thanks,
Eric

On Wed, Mar 16, 2016 at 8:44 AM, Edward Mikuszewski <emikuszewski(a)pivotal.io
wrote:
Great question, Dan. I'm curious to this as well - I just assumed it took
some time to report back the status.

On Wed, Mar 16, 2016 at 10:28 AM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

Quick question, when I push an app to CF I get this output at the bottom
after the app starts.

```
requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: sinatra-test-dan.cfapps.io
last uploaded: Wed Mar 16 14:54:50 UTC 2016
stack: unknown
buildpack: https://github.com/cloudfoundry/ruby-buildpack#v1.6.5

state since cpu memory disk
details
#0 running 2016-03-16 10:55:12 AM 0.1% 21M of 256M 68.3M of 1G
```

Anyone know why the stack always reports "unknown"? If I run `cf app`
right after, it will report the stack properly. It only shows up this way
on the initial push.

Versions:
- cf version 6.16.1+924508c-2016-02-26
- cf v231
- diego 0.1454.0
- garden 0.333.0

Thanks,

Dan


--
Ed Mikuszewski *| * <http://www.pivotal.io> *|* Mobile: 646-628-2872
*| *Email: emikuszewski(a)pivotal.io


Re: cf push "stack: unknown"

Edward Mikuszewski <emikuszewski@...>
 

Great question, Dan. I'm curious to this as well - I just assumed it took
some time to report back the status.

On Wed, Mar 16, 2016 at 10:28 AM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

Quick question, when I push an app to CF I get this output at the bottom
after the app starts.

```
requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: sinatra-test-dan.cfapps.io
last uploaded: Wed Mar 16 14:54:50 UTC 2016
stack: unknown
buildpack: https://github.com/cloudfoundry/ruby-buildpack#v1.6.5

state since cpu memory disk
details
#0 running 2016-03-16 10:55:12 AM 0.1% 21M of 256M 68.3M of 1G
```

Anyone know why the stack always reports "unknown"? If I run `cf app`
right after, it will report the stack properly. It only shows up this way
on the initial push.

Versions:
- cf version 6.16.1+924508c-2016-02-26
- cf v231
- diego 0.1454.0
- garden 0.333.0

Thanks,

Dan
--
Ed Mikuszewski *| * <http://www.pivotal.io> *|* Mobile: 646-628-2872 *| *
Email: emikuszewski(a)pivotal.io


cf push "stack: unknown"

Daniel Mikusa
 

Quick question, when I push an app to CF I get this output at the bottom
after the app starts.

```
requested state: started
instances: 1/1
usage: 256M x 1 instances
urls: sinatra-test-dan.cfapps.io
last uploaded: Wed Mar 16 14:54:50 UTC 2016
stack: unknown
buildpack: https://github.com/cloudfoundry/ruby-buildpack#v1.6.5

state since cpu memory disk
details
#0 running 2016-03-16 10:55:12 AM 0.1% 21M of 256M 68.3M of 1G
```

Anyone know why the stack always reports "unknown"? If I run `cf app`
right after, it will report the stack properly. It only shows up this way
on the initial push.

Versions:
- cf version 6.16.1+924508c-2016-02-26
- cf v231
- diego 0.1454.0
- garden 0.333.0

Thanks,

Dan


Re: cloud_connector_ng not starting on cf 231

Christian Schwerdtfeger
 

I have further information the to 403 error. internal_error.log states:

"2016/03/16 12:05:00 [error] 20579#0: *105 directory index of "/var/vcap/store/shared/" is forbidden, client: 10.10.0.104, server: blobstore.service.cf.internal, request: "GET / HTTP/1.1", host: "blobstore.service.cf.internal""

where the 10.10.0.104 is the ip of my api_z1 machine running the cloud_controller job. So this seems to be the cause of the 403 errors... any ideas?
Directory "/var/vcap/store/shared/" has owner "vcap:vcap" and the nginx worker processes are running as user vcap too.


cloud_connector_ng not starting on cf 231

Christian Schwerdtfeger
 

Hello,

I am facing some problems with the cloud_connector on cf-release 231.
I suspect the problem has something to do with the blobstore configuration, but I can't figure out what's wrong.

I want to use the WebDAV blobstore introduced with 231 and followed the instructions in the GoogleDocs document linked in the release notes, except that I don't want to migrate, so i removed the nfs-job completely and use the blobstore_z1 job instead.

You can find the relevant configuration options in my stub.yml (which i use with the manifest generation tooling in the repo for openstack ) at the end of this post after some questions which came up while debugging:

1. In the GoogleDocs document the "fog_connection" is set to null. If i do this, my cloud_controller emits errors, that this property should be a not null. So is the doc correct here? What am I doing wrong here?

2. The blobstore nginx doesn't succeed to start up because of some server_name_hash sizes. I fixed this by adding "server_names_hash_bucket_size 64;" manually in the conf. Is there a way to prevent from this error?

3. I found, that on the blobstore nginx, requests are coming in if I leave the fog_connection as "{"provider":"Local","local_root":"/var/vcap/nfs/shared"}". But all of the requests are rejected with a 403 status code. Any idea where this could come from?

Thank you very much.

properties:
cc:
buildpacks:
blobstore_type: webdav
webdav_config:
username: blobstore-username
password: blobstore-password
private_endpoint: http://blobstore.service.cf.internal
public_endpoint: https://blobstore.cloudfoundry.mydomain.com
fog_connection: null
cdn: null

droplets:
blobstore_type: webdav
webdav_config:
username: blobstore-username
password: blobstore-password
private_endpoint: http://blobstore.service.cf.internal
public_endpoint: https://blobstore.cloudfoundry.mydomain.com
fog_connection: null
cdn: null

packages:
blobstore_type: webdav
webdav_config:
username: blobstore-username
password: blobstore-password
private_endpoint: http://blobstore.service.cf.internal
public_endpoint: https://blobstore.cloudfoundry.mydomain.com
fog_connection: null
cdn: null

resource_pool:
blobstore_type: webdav
webdav_config:
username: blobstore-username
password: blobstore-password
private_endpoint: http://blobstore.service.cf.internal
public_endpoint: https://blobstore.cloudfoundry.mydomain.com
fog_connection: null
cdn: null

blobstore:
admin_users:
- username: blobstore-username
password: blobstore-password
port: 80
secure_link:
secret: blobstore-secret
tls:
cert: BLOBSTORE_CERT
port: 443
private_key: BLOBSTORE_PRIVATE_KEY

nfs_server:
address: null

jobs:
- name: api_z1
templates:
- name: consul_agent
release: cf
- name: go-buildpack
release: cf
- name: binary-buildpack
release: cf
- name: nodejs-buildpack
release: cf
- name: ruby-buildpack
release: cf
- name: php-buildpack
release: cf
- name: python-buildpack
release: cf
- name: staticfile-buildpack
release: cf
- name: cloud_controller_ng
release: cf
- name: cloud_controller_clock
release: cf
- name: cloud_controller_worker
release: cf
- name: metron_agent
release: cf
- name: statsd-injector
release: cf
- name: route_registrar
release: cf
properties:
nfs_server:
address: null
- name: nfs_z1
instances: 0
- name: blobstore_z1
instances: 1


Re: [uaa] cannot retrieve username with scim.userids scope

Yitao Jiang
 

Hi, Fillip,

here's decoded token

{
"jti": "6a0fc594-3db9-4a10-b720-63c47ff82ef9",
"sub": "48981ed6-c72f-4072-939c-ad7a4e36a669",
"scope": [
"cloud_controller.read",
"password.write",
"cloud_controller.write",
"openid",
"uaa.user"
],
"client_id": "myconsole",
"cid": "myconsole",
"azp": "myconsole",
"grant_type": "password",
"user_id": "48981ed6-c72f-4072-939c-ad7a4e36a669",
"origin": "uaa",
"user_name": "XXX",
"email": "XXX",
"auth_time": 1458092554,
"rev_sig": "5aef8031",
"iat": 1458092554,
"exp": 1458135754,
"iss": "http://uaa.XXX.com/oauth/token",
"zid": "uaa",
"aud": [
"myconsole",
"cloud_controller",
"password",
"openid",
"uaa"
]
}

doesn't contain scim scope

On Wed, Mar 16, 2016 at 1:07 AM, Filip Hanik <fhanik(a)pivotal.io> wrote:

Take a look at the value of $TOKEN (many online decoders out there.
https://jwt.io is one) and see what scopes your token actually has.

Filip

On Tue, Mar 15, 2016 at 8:45 AM, Yitao Jiang <jiangyt.cn(a)gmail.com> wrote:

Hi, guys,

I wanna get the users email , so per the docs of UAA at
https://github.com/cloudfoundry/uaa/blob/master/docs/UAA-APIs.rst#query-for-information-get-users,
i create a client with following scopes, scim.userids cloud_controller.read
password.write cloud_controller.write openid scim.write scim.read
cloud_controller.admin and with grant types:
authorization_code,refresh_token,client_credentials,password

when using this client to login a user , the JWT of the token parsed
doesn't contain scim.read scopt, lead to fail calling /Users api.
But , when login the client using uaac and using uaac context to obtain
the token, the token has scim.read scope and success calling /Users api

Here's related infos

​#​
uaac client get myconsole

scope: cloud_controller.admin cloud_controller.read
cloud_controller.write openid password.write
​ ​
scim.read scim.userids scim.write uaa.user
client_id: myconsole
resource_ids: none
authorized_grant_types: authorization_code client_credentials password
refresh_token
autoapprove: true
action: none
authorities: scim.userids cloud_controller.read password.write
cloud_controller.write openid
​ ​
scim.write scim.read cloud_controller.admin
name: myconsole
lastmodified: 1458017396000

​login the user user1 using myconsole client​

curl -X POST -d"username=
​user1(a)abc.com​
&password=password&client_id=myconsole&client_secret=
​XXX
&grant_type=password" -u "myconsole:
​XXX
" http://uaa.
​XXX
.com/oauth/token

got the token
get the users

curl -v -X GET -H "Accept: application/json" -H "Authorization: basic
$TOKEN" http://uaa.
​XXX.
com/Users?attributes=userName​

failed with

{
"error": "insufficient_scope",
"error_description": "Insufficient scope for this resource",
"scope": "scim.read zones.uaa.admin"
}​


​But if replace token with uaac context returned, i could get the users​




--

Regards,

Yitao

--

Regards,

Yitao


Re: Dial tcp: i/o timeout while pushing a sample app to Cloud Foundry BOSH-Lite deployment

Giovanni Napoli
 

Hello Rob, sure...i will post to the other thread. Thank you for your answer.


Re: NFS

Ted Young
 

Hi Raymond,

It is possible to use FUSE-based clients to connect to remote filesystems,
but only if your Cloud Foundry deployment has enabled privileged containers.

We are currently working on adding volume management to Cloud Foundry. NFS
and other distributed filesystems will be available as services, allowing
access without the need for privileged containers or FUSE.

Thanks,
Ted

On Tue, Mar 15, 2016 at 1:21 PM, Raymond J Steele <raymondsteele(a)gmail.com>
wrote:

Can a cloud foundry application make us of Network Filesystem share? If
so, would this scale?

Thanks,

Raymond

5201 - 5220 of 9425