Date   

future changes to etcd configuration in cf-release

Amit Kumar Gupta
 

Hi all,

Just wanted to give the community advance notice that we will be
introducing a change to the etcd configuration in cf-release, probably
within the week (probably cf v220+, we are currently on v218).

etcd can be configured to require ssl communication amongst servers, and
between servers and clients. Currently this defaults to false, but we will
be changing the default to true. We will include documentation on how to
generate certs, and where to put them in your stubs if you are using the
spiff tooling to generate deployment manifests. The BOSH-Lite dev
manifests will include certs by default, to make the dev workflow
especially easy.

Cheers,

Amit Gupta
Cloud Foundry PM, OSS Release Integration team


proposed stemcell network performance tuning

Amit Kumar Gupta
 

Hi all,

I'd like to propose tuning a couple kernel parameters related to tcp
performance:

# TCP_FIN_TIMEOUT
# This setting determines the time that must elapse before TCP/IP can
release a closed connection and reuse
# its resources. During this TIME_WAIT state, reopening the connection to
the client costs less than establishing
# a new connection. By reducing the value of this entry, TCP/IP can release
closed connections faster, making more
# resources available for new connections. Adjust this in the presence of
many connections sitting in the
# TIME_WAIT state:

echo 5 > /proc/sys/net/ipv4/tcp_fin_timeout

# TCP_TW_REUSE
# This allows reusing sockets in TIME_WAIT state for new connections when
it is safe from protocol viewpoint.
# Default value is 0 (disabled). It is generally a safer alternative to
tcp_tw_recycle

echo 1 > /proc/sys/net/ipv4/tcp_tw_reuse

Currently, these parameters are set by certain jobs in cf-release,
diego-release, and perhaps others. Any VM needing to establish a high
number of incoming/outgoing tcp connections in a short period of time will
be unable to establish new connections without changing these parameters.

We believe these parameters are safe to change across the board, and will
be generally beneficial. The existing defaults made sense for much older
networks, but can be greatly optimized for modern systems.

Please share with the mailing lists if you have any questions or feedback
about this proposal. If you maintain a bosh release and would like to see
how these changes would affect your release, you can create a job which
simply does the above in its startup scripts, and colocate that job with
all the other jobs in a deployment of your release.

Thanks,

Amit Gupta
Cloud Foundry PM, OSS Release Integration team


Magnesium Metal, Ferro Phosphorus, Calcium Ferrite

Alexander Yang <alexanderjoe2@...>
 

Dear Sir,
 
Currently, we could supply you with the cargo as below.
 
1. Magnesium ingot,alloy and powder
 
2. Ferro Phosphorus
 
3. Calcium Ferrite
 
 
If you have demand, please let me know.
 
Yours sincerely,
Alexander Yang
 
 
++++++++++++++++++++++++++++++++++++++++++++++++++++
 

Mobile      : +86-18608554433
 
Tel           : +86-855- 8589919
Fax          : +86-855-8589919
 
 
Email        : alexanderjoe6@... d.suns@...
 
Skype       : dualsuns1
 
++++++++++++++++++++++++++++++++++++++++++++++++++++
 


Re: CF API for "general statistics"?

CF Runtime
 

I'd like to add as well that you can configure Bosh's healthmonitor to send
this data to an external dataviz/archival service (we use Datadog for our
various test environments): https://bosh.io/docs/hm-config.html

Zak + Mikhail, CF Release Integration

On Tue, Sep 29, 2015 at 4:58 AM, James Bayer <jbayer(a)pivotal.io> wrote:

if you're using bosh to manage cloud foundry, then the bosh cli has a
command:
bosh vms --vitals

which you can read about here [1], that does a good job of measuring infra
concerns:

- Vitals: Includes load, CPU, memory, swap, system disk, ephemeral
disk, and persistent disk usage for each VM.


[1] http://bosh.io/docs/sysadmin-commands.html#health

On Tue, Sep 29, 2015 at 2:51 AM, Rafal Radecki <radecki.rafal(a)gmail.com>
wrote:

Hi all :)

I got a task to check the possibility to fetch details such as:
- overall used and available memory/storage usage of cf platform
- list of running warden containers, approximation how many can be
spawned additionally
I looked through https://apidocs.cloudfoundry.org/218/ but did not find
anything similar to whats I am looking for.

Are there any options other than standard os level monitoring?

BR,
Rafal.


--
Thank you,

James Bayer


Re: Making your landscape trust a certain certificate authority

Daniel Mikusa
 

A couple options...

1.) If you only need the cert for one or a couple apps and / or you don't
want to manage a fork, you can package the CA cert with the application.
Then add a `.profile.d` script [1] that sets up the certificate that you
packaged with the app. I don't have an example off-hand, but the
.profile.d script could call keytool and import your ca cert or just move a
keystore containing your ca cert from the app files and overwrite the JVM
default.

2.) Write your application so that it can utilize it's own truststore.
It's a little more work, but most things that use a truststore in the JVM
allow you to customize the behavior specifying your own truststore. If the
app were to do this, it could then expose a configuration option through
which it's users could set their own truststore. Then users wouldn't need
to depend on what certs are installed by default.

Dan


[1] - https://devcenter.heroku.com/articles/profiled (CF honors them too)

On Tue, Sep 29, 2015 at 4:50 PM, Mathias Essenpreis <esse(a)gmx.de> wrote:

Hi,
we want to make the apps on our landscape trust a certain ca certificate
of a ca we run internally. I came across the instructions on how to do this
for the JRE buildpack:
https://github.com/cloudfoundry/java-buildpack/blob/master/docs/jre-open_jdk_jre.md#custom-ca-certificates

I understand doing this once in the JRE buildpack will directly make all
the java apps that of course need to make use of the JRE buildpack trusting
our internal ca. Is that correct or do I need to change other buildpacks as
well?

However this implies performing a fork of the JRE buildpack. A fork means
that we always need to synchronize manually with the newer versions of the
build pack.

How do you approach this problem in your organization? I suppose you also
have internal certificate authorities that need to be trusted, right?

Also I am asking myself whether there is a way to enhance the default
buildpack allowing an application (instead of the buildpack) to bring their
own, additional certificates?

thx
Mathias


Making your landscape trust a certain certificate authority

Mathias Essenpreis <esse@...>
 

Hi,
we want to make the apps on our landscape trust a certain ca certificate of a ca we run internally. I came across the instructions on how to do this for the JRE buildpack: https://github.com/cloudfoundry/java-buildpack/blob/master/docs/jre-open_jdk_jre.md#custom-ca-certificates

I understand doing this once in the JRE buildpack will directly make all the java apps that of course need to make use of the JRE buildpack trusting our internal ca. Is that correct or do I need to change other buildpacks as well?

However this implies performing a fork of the JRE buildpack. A fork means that we always need to synchronize manually with the newer versions of the build pack.

How do you approach this problem in your organization? I suppose you also have internal certificate authorities that need to be trusted, right?

Also I am asking myself whether there is a way to enhance the default buildpack allowing an application (instead of the buildpack) to bring their own, additional certificates?

thx
Mathias


Re: cloud_controller_ng performance degrades slowly over time

Mike Youngstrom <youngm@...>
 

We don't have the same performance numbers since we don't use New Relic.
However, for the last 6 months or so after 3-4 weeks our smoke tests begin
failing randomly with 500 errors doing basic things like creating routes.
If we restart the CC(s) then everything works great for another 3-4 weeks.

Mike

On Tue, Sep 29, 2015 at 1:08 PM, Matt Cholick <cholick(a)gmail.com> wrote:

This is a pretty tricky one, as it takes a long time to manifest. After a
while without a restart, cloud_controller_ng take a long time listing org
users. For example, in an org with 350 users, before restart `cf org-users
ORG` took 1:20. After restart, the call took 0:07. We've only notice this
twice, both times after the cloud controller had been running for a couple
weeks without a restart.

Looking in New Relic, the breakdown of an individual call to
organization/guid/managers shows the majority of the call externally:

[image: Inline image 1]

Though uaa itself isn't the issue, as things are fine immediately after
restarting cloud_controller_ng (and restarting uaa has no affect).

Have other Cloud Foundry operators seen this degraded performance? Is
there other information we could provide to turn this into a workable bug
report?

-Matt Cholick



cloud_controller_ng performance degrades slowly over time

Matt Cholick
 

This is a pretty tricky one, as it takes a long time to manifest. After a
while without a restart, cloud_controller_ng take a long time listing org
users. For example, in an org with 350 users, before restart `cf org-users
ORG` took 1:20. After restart, the call took 0:07. We've only notice this
twice, both times after the cloud controller had been running for a couple
weeks without a restart.

Looking in New Relic, the breakdown of an individual call to
organization/guid/managers shows the majority of the call externally:

[image: Inline image 1]

Though uaa itself isn't the issue, as things are fine immediately after
restarting cloud_controller_ng (and restarting uaa has no affect).

Have other Cloud Foundry operators seen this degraded performance? Is there
other information we could provide to turn this into a workable bug report?

-Matt Cholick


Re: [ann] Subway - how to scale out any Cloud Foundry service

Dr Nic Williams
 

Subway is now also available as a bosh release; and works nicely collocated on a now-scalable bosh deployment https://blog.starkandwayne.com/2015/09/29/deploying-subway-broker-with-bosh/

On Mon, Sep 21, 2015 at 4:27 PM, Dr Nic Williams <drnic(a)starkandwayne.com>
wrote:

Quicky links:
* https://github.com/cloudfoundry-community/cf-subway
*
https://blog.starkandwayne.com/2015/09/21/how-to-scale-out-any-cloud-foundry-service/
We've been using Ferdy's Docker BOSH release since he created it, and have
published new docker images, new wrapper BOSH releases and more. But it
still doesn't scale horizontally (yes it has docker swarm support but no
that can't do persistent storage on volumes).
So we created Subway - a broker that allows you to run a fleet of
single-server service brokers such as Docker BOSH release, or
cf-redis-boshrelease.
I'll write up/create a video soon to walk-thru upgrading your existing
in-production single-server services to use Subway.
Have fun!
Nic
--
Dr Nic Williams
Stark & Wayne LLC - consultancy for Cloud Foundry users
http://drnicwilliams.com
http://starkandwayne.com
cell +1 (415) 860-2185
twitter @drnic


Using the cf-dev list for CF Abacus dev discussions

Jean-Sebastien Delfino
 

Hi all,

We've been making pretty good progress with the CF Abacus (incubating)
project recently.

So far, we've been using a mix of Gitter, Slack and Github issues for all
our detailed dev discussions but with the increased momentum in the project
I'm thinking that it's probably be a good time to start moving these dev
discussions to the cf-dev list instead.

I'm hoping that the mailing list will help organize our Abacus dev
discussions a bit better, and it should give others on the list more
opportunities to follow and participate as well.

If there's no objection I'll start posting here in a day or so.

-- Jean-Sebastien


Re: Permission denied error when unpacking droplet

Rohit Kelapure <rkelapure@...>
 

On Mon, Sep 28, 2015 at 7:03 PM, Amit Gupta <amitkgupta84(a)gmail.com> wrote:

The DEA is trying to run a script in the warden container to untar the
droplet. For some reason, the user executing the script and the file
system permissions in the container in the /home/vcap directory are not
what they're supposed to be. I believe you are deploying and running all
the Cloud Foundry components in a non-standard way. You may wish to try a
more standard deployment, and look at the differences in users and file
system permissions between the standard deployment and your deployment, and
reverse-engineer what would need to be fixed in your deployment to get
things to work. Here's a blog post, at the end it talks about how you can
get a session into a warden container (
http://andypiper.co.uk/2014/02/17/getting-inside-cloud-foundry-for-debug-and-profit/).
Alternatively you can look at the startup scripts used in the standard
deployments, e.g. (
https://github.com/cloudfoundry/cf-release/blob/master/jobs/dea_next/templates/dea_ctl.erb
).

Since you've configured all your components in a very idiosyncratic
manner, it's hard to say generally what you need to fix to get all the
permissions right.
--
Pivotal CF Architect
rkelapure(a)pivotal.io
cell: 9197242524


Re: PHP extension 'gettext' doesn't work?

Daniel Mikusa
 

It seems to be working for me [1] (at least the extension is loading, which
usually means it'll work). Can you provide some details on what you're
seeing?

- what version of PHP are you using?
- how are you enabling the extension: options.json or composer.json?
- what's the full output of `cf push`?
- what's the output of `cf logs` when you try to access a page that uses
`gettext`?

Thanks,

Dan

[1] - http://php-info.cfapps.io/info.php

On Tue, Sep 29, 2015 at 3:56 AM, Hiroaki Ukaji <dt3snow.w(a)gmail.com> wrote:

Hi.

We deployed a php application using 'gettext' extension to manage i18n
function. (*1)

We tried to activate i18n, however, it didn't work in this php application
after all.
We also tested with a simple application which contains 'gettext', but it
seemed that the extension still doesn't work.(*2)

Is there anyone who uses i18n well by using 'gettext' extension in PHP
applications?


----------
(*1)
http://blog.cloudfoundry.gr.jp/2015/09/cf100apps-067-fc2blog.html
----------
(*2)

This result should be `/home/vcap/app/htdocs/locale: Hello, world` , if
'gettext' extension works well.
----------

Thanks.

Hiroaki UKAJI




--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-PHP-extension-gettext-doesn-t-work-tp1984.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: CF API for "general statistics"?

James Bayer
 

if you're using bosh to manage cloud foundry, then the bosh cli has a
command:
bosh vms --vitals

which you can read about here [1], that does a good job of measuring infra
concerns:

- Vitals: Includes load, CPU, memory, swap, system disk, ephemeral disk,
and persistent disk usage for each VM.


[1] http://bosh.io/docs/sysadmin-commands.html#health

On Tue, Sep 29, 2015 at 2:51 AM, Rafal Radecki <radecki.rafal(a)gmail.com>
wrote:

Hi all :)

I got a task to check the possibility to fetch details such as:
- overall used and available memory/storage usage of cf platform
- list of running warden containers, approximation how many can be spawned
additionally
I looked through https://apidocs.cloudfoundry.org/218/ but did not find
anything similar to whats I am looking for.

Are there any options other than standard os level monitoring?

BR,
Rafal.


--
Thank you,

James Bayer


CF API for "general statistics"?

Rafal Radecki
 

Hi all :)

I got a task to check the possibility to fetch details such as:
- overall used and available memory/storage usage of cf platform
- list of running warden containers, approximation how many can be spawned additionally
I looked through https://apidocs.cloudfoundry.org/218/ but did not find anything similar to whats I am looking for.

Are there any options other than standard os level monitoring?

BR,
Rafal.


PHP extension 'gettext' doesn't work?

Hiroaki Ukaji <dt3snow.w@...>
 

Hi.

We deployed a php application using 'gettext' extension to manage i18n
function. (*1)

We tried to activate i18n, however, it didn't work in this php application
after all.
We also tested with a simple application which contains 'gettext', but it
seemed that the extension still doesn't work.(*2)

Is there anyone who uses i18n well by using 'gettext' extension in PHP
applications?


----------
(*1)
http://blog.cloudfoundry.gr.jp/2015/09/cf100apps-067-fc2blog.html
----------
(*2)

This result should be `/home/vcap/app/htdocs/locale: Hello, world` , if
'gettext' extension works well.
----------

Thanks.

Hiroaki UKAJI




--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-PHP-extension-gettext-doesn-t-work-tp1984.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: Permission denied error when unpacking droplet

Amit Gupta
 

The DEA is trying to run a script in the warden container to untar the droplet. For some reason, the user executing the script and the file system permissions in the container in the /home/vcap directory are not what they're supposed to be. I believe you are deploying and running all the Cloud Foundry components in a non-standard way. You may wish to try a more standard deployment, and look at the differences in users and file system permissions between the standard deployment and your deployment, and reverse-engineer what would need to be fixed in your deployment to get things to work. Here's a blog post, at the end it talks about how you can get a session into a warden container (http://andypiper.co.uk/2014/02/17/getting-inside-cloud-foundry-for-debug-and-profit/). Alternatively you can look at the startup scripts used in the standard deployments, e.g. (https://github.com/cloudfoundry/cf-release/blob/master/jobs/dea_next/templates/dea_ctl.erb).

Since you've configured all your components in a very idiosyncratic manner, it's hard to say generally what you need to fix to get all the permissions right.


Re: my app needs to get the number of instances in which its running, (in runtime my app uses this info in my program logic)

CF Runtime
 

This information is not provided to the app. You can see what data is
provided here:
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#instance-vars

If you want to get this information for you app, the best way would be to
have the app call the API directly and see what the "instances" attribute
is set to.
http://apidocs.cloudfoundry.org/218/apps/retrieve_a_particular_app.html

Joseph
CF Release Integration Team

On Mon, Sep 28, 2015 at 12:14 PM, zooba Sir <myfakename90(a)gmail.com> wrote:

evaluating the performance of my app.
how my app can get the number of instances in which its running. in
runtime it needs this info in my logic.


my app needs to get the number of instances in which its running, (in runtime my app uses this info in my program logic)

Zuba Al <myfakename90@...>
 

evaluating the performance of my app.
how my app can get the number of instances in which its running. in runtime it needs this info in my logic.


Re: Instance crashing after running once. Error: "reason"=>"CRASHED", "exit_status"=>0, "exit_description"=>"app instance exited"

Zuba Al <myfakename90@...>
 

i thought exit with status 0 is error. my app is running good then.
thanks Dan.


Re: postgres out of disk space

Matthias Ender <Matthias.Ender@...>
 

thank you all!
I’m indeed running v212.
Deleting /var/vcap/store/shared/run.10.10.2.37.xip.io-cc-droplets/buildpack_cache
did the trick.


From: CF Runtime [mailto:cfruntime(a)gmail.com]
Sent: Friday, September 25, 2015 6:47 PM
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] Re: Re: Re: postgres out of disk space
Importance: High

cf-release v213 fixed a bug where buildpack caches were not getting cleaned up properly. Are you running a version prior to 213?

If so, you might need to go into the `shared/run.10.10.2.37.xip.io-cc-droplets` directory and delete the `buildpack_cache` subdirectory. Deleting the contents of buildpack_cache will not cause failures in the system, the caches will simply get generated from scratch the next time apps stage.

If you upgrade to 213 or later, you can just call the api endpoint to clean out the buildpack cache. You can find the details in the v213 release notes:
https://github.com/cloudfoundry/cf-release/releases/tag/v213

Joseph & Natalie
CF Release Integration Team

On Fri, Sep 25, 2015 at 11:53 AM, Matthias Ender <Matthias.Ender(a)sas.com<mailto:Matthias.Ender(a)sas.com>> wrote:
ah, it’s not postgres:

90M postgres

it’s actually:

94G shared/run.10.10.2.37.xip.io-cc-droplets

Does it keep a copy of each artifact ever pushed?
Or is this part of the event storage, controlled by these properties:

app_events:
cutoff_age_in_days: 31
app_usage_events:
cutoff_age_in_days: 31
audit_events:
cutoff_age_in_days: 31

??


From: Aleksey Zalesov [mailto:aleksey.zalesov(a)altoros.com<mailto:aleksey.zalesov(a)altoros.com>]
Sent: Friday, September 25, 2015 12:23 PM
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>>
Subject: [cf-dev] Re: postgres out of disk space

Does your database occupy this 100 GB? Or something else like logs?

Aleksey Zalesov | CloudFoundry Engineer | Altoros
Tel: (617) 841-2121 ext. 5707<tel:%28617%29%20841-2121%20ext.%205707> | Toll free: 855-ALTOROS
Fax: (866) 201-3646<tel:%28866%29%20201-3646> | Skype: aleksey_zalesov
www.altoros.com<http://www.altoros.com/> | blog.altoros.com<http://blog.altoros.com/> | twitter.com/altoros<http://twitter.com/altoros>

On 25 Sep 2015, at 14:22, Matthias Ender <Matthias.Ender(a)sas.com<mailto:Matthias.Ender(a)sas.com>> wrote:

I have a cf-aws-tiny cf-boshrelease deployment, and it’s been running well for over 4 months.
We have about 40 apps, with a couple of dozen of cf pushes each day.
Yesterday pushing apps became spotty and then impossible, with various errors.
Turned out the 100GB disk for the postgres instance on the data note was full.
I increased the disk size and things a running again.
But – what happened there? 100G and growing seems like awfully large database for a rather modest use.
And I’m worried it’ll just happen again in a few months.

thanks,
Matthias