Date   

Re: Request timeout in CloudFoundry

Paul Bakare
 

Hi,

I'm experiencing this same issue.

I have a NodeJS app, using the nodejs buildpack. A request taking
approximately 80 secs long gets timed out (502 response). We're using the
default CF ELBs (gorouter and ha_proxy).

On Tue, Aug 25, 2015 at 10:57 PM, CF Runtime <cfruntime(a)gmail.com> wrote:

Hi,

We think this message was dropped and so we are posting it again.

How long of a timeout are you seeing? The gorouter and ha_proxy have a
default timeout of 15 minutes. If you are using an ELB or some other load
balancer they may have their own timeout set. I'm not sure if any of the
app servers installed by buildpacks have a build in timeout or not, what
type of app is it?

Joseph and Dan
OSS Release Integration Team

On Mon, Aug 17, 2015 at 2:18 PM, Flávio Henrique Schuindt da Silva <
flavio.schuindt(a)gmail.com> wrote:

Hi guys,

How can I increase the request timeout in CF? I would like to give more
time to my app to send a response to the requests before getting a timeout.
Should I increase it in the buildpack? If yes, where should I change it on
java buildpack?

Thanks in advance!


Re: Correcting CF Docs

Amit Kumar Gupta
 

Hey Daniel,

I'll let the documentation team know about this issue.

Joaquin, thanks! I did not know that.

Best,
Amit, CF Release Integration PM

On Thu, Sep 3, 2015 at 12:30 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Thanks Joaquin!

I found out the long way that just setting the region will do. Sadly last
time I used a light stemcell non-US-east regions weren't supported. Given
that the documentation suggested that there was still a separate process, I
wasted some time with a full-fat stemcell that didn't work.

Bit of a shame as I was trying to demonstrate how easy Cloud Foundry was
to install on AWS to a client!

Does anyone know who owns the documentation or how to update it?

On Wed, Sep 2, 2015 at 9:17 PM, Joaquin Zalazar <
joaquin.zalazar(a)altoros.com> wrote:

Hi Daniel!

What I've done while testing is to set the variable BOSH_AWS_REGION to
the region I needed.

export BOSH_AWS_REGION=<your region here>

Regards!!

Joaquin Zalazar

Altoros


On Wed, Sep 2, 2015 at 9:43 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Hi all,

The CF docs for deploying CF to an AWS VPC
<https://docs.cloudfoundry.org/deploying/ec2/bootstrap-aws-vpc.html>
have a broken link to a Gist of Dan Highham's, that presumably talks about
deploying to regions other than us-east-1.

Does anyone know where the instructions for non-US regions are now?

Regards,

Daniel Jones
EngineerBetter.com


--

*Joaquin Zalazar*
*ALTOROS* — Cloud Foundry deployment, training and integration
www.altoros.com


--
Regards,

Daniel Jones
EngineerBetter.com


Re: stdout.log and stderr.log not show in CF197 with loggregator enabled

ramonskie
 

the only option here is to route the log output to something like graylog or logstash

and to my knowledge what i understood from release 215 you can set a longer retention log for the --recent log options
but i'm not sure

-----Original Message-----
From: "Ravindra, Shruthi (GE Healthcare)" <Shruthi.Ravindra(a)ge.com<mailto:%22Ravindra,%20Shruthi%20%28GE%20Healthcare%29%22%20%3cShruthi.Ravindra(a)ge.com%3e>>
Reply-to: Discussions about Cloud Foundry projects and the system overall. <cf-dev(a)lists.cloudfoundry.org>
To: cf-dev(a)lists.cloudfoundry.org <cf-dev(a)lists.cloudfoundry.org<mailto:%22cf-dev(a)lists.cloudfoundry.org%22%20%3ccf-dev(a)lists.cloudfoundry.org%3e>>
Subject: [cf-dev] Re: stdout.log and stderr.log not show in CF197 with loggregator enabled
Date: Wed, 2 Sep 2015 13:04:18 +0000

Hi ,



We are having the same problem where we are not able to see the stdout.log and stderror.log for all the application deployed in the cloud.

Could you please let me know how do we access the logs other than cf logs “appname” command which is not giving us the full logs.



Thanks,

Shruthi



********************************************************
For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message.

Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt.
Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286
********************************************************


Re: Correcting CF Docs

Daniel Jones
 

Thanks Joaquin!

I found out the long way that just setting the region will do. Sadly last
time I used a light stemcell non-US-east regions weren't supported. Given
that the documentation suggested that there was still a separate process, I
wasted some time with a full-fat stemcell that didn't work.

Bit of a shame as I was trying to demonstrate how easy Cloud Foundry was to
install on AWS to a client!

Does anyone know who owns the documentation or how to update it?

On Wed, Sep 2, 2015 at 9:17 PM, Joaquin Zalazar <joaquin.zalazar(a)altoros.com
wrote:
Hi Daniel!

What I've done while testing is to set the variable BOSH_AWS_REGION to
the region I needed.

export BOSH_AWS_REGION=<your region here>

Regards!!

Joaquin Zalazar

Altoros


On Wed, Sep 2, 2015 at 9:43 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Hi all,

The CF docs for deploying CF to an AWS VPC
<https://docs.cloudfoundry.org/deploying/ec2/bootstrap-aws-vpc.html>
have a broken link to a Gist of Dan Highham's, that presumably talks about
deploying to regions other than us-east-1.

Does anyone know where the instructions for non-US regions are now?

Regards,

Daniel Jones
EngineerBetter.com


--

*Joaquin Zalazar*
*ALTOROS* — Cloud Foundry deployment, training and integration
www.altoros.com
--
Regards,

Daniel Jones
EngineerBetter.com


Re: diego questions: default memory and passing env variables

Rohit Kelapure <rkelapure@...>
 

Setting the JBP_CONFIG_OPEN_JDK_JRE only works in JBP ver. 3.0.0.+
see
http://chrisfrost.com/cloudfoundry/configuring-the-cloud-foundry-java-buildpack/
for some context.

-cheers,
Rohit

On Wed, Sep 2, 2015 at 8:55 PM, Rohit Kelapure <rkelapure(a)pivotal.io> wrote:

To add more native memory to the JVM provisioned by the buildpack, you
need to change the memory heuristics of the algorithm that generates the
JVM -X memory settings. see
https://github.com/pivotalservices/citigroup/blob/master/java-buildpack/docs/jre-open_jdk_jre.md

One way to increase the amount of native memory and decrease the memory
allocated to the JVM heap is by setting the following env variable

cf set-env spring-music JBP_CONFIG_OPEN_JDK_JRE '[memory_heuristics:
{heap: 65, metaspace: 20}, memory_sizes: {metaspace: 256m..}]'

Here we are changing the increasing the allocation of metaspace from its
default 64m to 256m in JDK 1.8.
This will give more headroom to the native memory. This is an alternative
to setting the MEMORY_LIMIT variable.

-cheers,
Rohit Kelapure


On Wed, Sep 2, 2015 at 8:47 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Tweaking the buildpack is less dynamic, and every time you want to make a
change you will have to tweak again, create the new buildpack, and upload
it to the CC. Moreover, maintaining your own fork of the buildpack adds
overhead when you need to consume important changes from the standard
buildpack, such as security updates.

On Wed, Sep 2, 2015 at 8:34 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Thanks for the clarification Sabha and Amit. We are using JavaBP and our
usecase is to have additional native memory headroom for the app. One
option is to tweak the open_jdk.yml file in the JavaBP to give more native
memory and reduce the heap in that process or to set this MEMORY_LIMIT env
variable. We are going with setting the MEMORY_LIMIT option.

Regards,
Siva

On Wed, Sep 2, 2015 at 5:35 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Ah, I was mistaken. If you set MEMORY_LIMIT, it will not get
overwritten. So if you don't set it, it will be set to the container
memory limit. But if you do set it, then the java buildpack can take
advantage of it in the way Sabha explained.

On Wed, Sep 2, 2015 at 5:18 PM, Sabha Parameswaran <sabhap(a)pivotal.io>
wrote:

The MEMORY_LIMIT is used by the java buildpack to configure the
various heap parameters for a java app. This variable value is set to the
container size by default by the Cloud controller when you push the app and
JBP uses that to arrive at heap during staging.

What you have done is making the buildpack work with a lower memory
threshold even though the app itself is allowed a higher limit.

Tweak the MEMORY_LIMIT variable only when you are using JavaBP and
want to really have some additional native memory headroom to grow (due to
native memory usage or safety concerns) or to limit heap from utilizing the
full container memory size.

As Amit mentioned, always use cf scale option to change the memory or
instance count for the app container.

-Sabha

On Wed, Sep 2, 2015 at 4:17 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Thanks for the response. But what I am observing is, by setting
MEMORY_LIMIT as an environment variable does take effect when I restage the
app and it reduces the memory allocated to the java app(not the container
itself) to the amount set in the variable. The app is restarted with the
same heap ratio as dictated by the buildpack but assumes the max memory
available to it is now 1GB instead of 2GB. The container still had 2GB
allocated to it and I understand that will be reduced only if I use the "cf
scale" command.
Is this the expected behavior?

Thanks
Siva

On Wed, Sep 2, 2015 at 3:55 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

This is not a environment variable you can set. To scale down the
memory allocated to your instances, do

cf scale <app-name> -m 1024M

This will tell Warden/Garden to allocate that much memory to your
containers and kill them if they exceed it. The documentation is saying
that the processes running inside the container can learn about the memory
limit applied to them via the environment variable MEMORY_LIMIT. If you
try to set this variable via set-env, it will get overwritten by the value
set by "cf scale". Again, the MEMORY_LIMIT environment variable doesn't do
anything, it's just information passed into the container via the process's
environment. To actually constrain the memory, use "cf scale" or set it in
your manifest file:


https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#memory

On Wed, Sep 2, 2015 at 3:08 PM, Siva Balan <mailsiva(a)gmail.com>
wrote:

We are not using Diego and we are on CF version 211.
I had my application started with 2GB of memory set in the manifest
file. I then ran the following command and restaged the app:
cf set-env <app-name> MEMORY_LIMIT 1024m
Then when I ran "cf app <app-name>", it was still showing the
instance memory of 2GB but the heap size allocated to the application was
cut in half and I confirmed this through New Relic app. Also, the first
full GC of the app occurred when the memory of the instance was at 1.3GB.
So I am not quite sure of the description of the MEMORY_LIMIT env variable
as described in
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#memory.
I did not see my application restart when the instance memory went over
1GB.
Any ideas about this behavior?

Thanks.
Siva

On Tue, Sep 1, 2015 at 7:40 PM, Amit Gupta <agupta(a)pivotal.io>
wrote:

Responses inline.

On Tue, Sep 1, 2015 at 7:13 PM, Tom Sherrod <tom.sherrod(a)gmail.com
wrote:
What is the default memory size of an app deployed in diego? cf
apps indicate 1GB.
Yes. This isn't a diego thing, this is a CC thing.


cf set-env doesn't appear to be getting the environment variable
information to the running container. Is cf set-env the correct method to
use?
You likely need to restage the application for ENV var changes to
take effect. Unless I'm mistaken, this too is not diego-specific, and
applies to DEA containers as well.


Thanks!
Tom
Best,
Amit

--
http://www.twitter.com/sivabalans

--
http://www.twitter.com/sivabalans

--
Pivotal CF Architect
rkelapure(a)pivotal.io
cell: 9197242524

--
Pivotal CF Architect
rkelapure(a)pivotal.io
cell: 9197242524


Re: diego questions: default memory and passing env variables

Rohit Kelapure <rkelapure@...>
 

To add more native memory to the JVM provisioned by the buildpack, you need
to change the memory heuristics of the algorithm that generates the JVM -X
memory settings. see
https://github.com/pivotalservices/citigroup/blob/master/java-buildpack/docs/jre-open_jdk_jre.md

One way to increase the amount of native memory and decrease the memory
allocated to the JVM heap is by setting the following env variable

cf set-env spring-music JBP_CONFIG_OPEN_JDK_JRE '[memory_heuristics: {heap:
65, metaspace: 20}, memory_sizes: {metaspace: 256m..}]'

Here we are changing the increasing the allocation of metaspace from its
default 64m to 256m in JDK 1.8.
This will give more headroom to the native memory. This is an alternative
to setting the MEMORY_LIMIT variable.

-cheers,
Rohit Kelapure

On Wed, Sep 2, 2015 at 8:47 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Tweaking the buildpack is less dynamic, and every time you want to make a
change you will have to tweak again, create the new buildpack, and upload
it to the CC. Moreover, maintaining your own fork of the buildpack adds
overhead when you need to consume important changes from the standard
buildpack, such as security updates.

On Wed, Sep 2, 2015 at 8:34 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Thanks for the clarification Sabha and Amit. We are using JavaBP and our
usecase is to have additional native memory headroom for the app. One
option is to tweak the open_jdk.yml file in the JavaBP to give more native
memory and reduce the heap in that process or to set this MEMORY_LIMIT env
variable. We are going with setting the MEMORY_LIMIT option.

Regards,
Siva

On Wed, Sep 2, 2015 at 5:35 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Ah, I was mistaken. If you set MEMORY_LIMIT, it will not get
overwritten. So if you don't set it, it will be set to the container
memory limit. But if you do set it, then the java buildpack can take
advantage of it in the way Sabha explained.

On Wed, Sep 2, 2015 at 5:18 PM, Sabha Parameswaran <sabhap(a)pivotal.io>
wrote:

The MEMORY_LIMIT is used by the java buildpack to configure the various
heap parameters for a java app. This variable value is set to the container
size by default by the Cloud controller when you push the app and JBP uses
that to arrive at heap during staging.

What you have done is making the buildpack work with a lower memory
threshold even though the app itself is allowed a higher limit.

Tweak the MEMORY_LIMIT variable only when you are using JavaBP and want
to really have some additional native memory headroom to grow (due to
native memory usage or safety concerns) or to limit heap from utilizing the
full container memory size.

As Amit mentioned, always use cf scale option to change the memory or
instance count for the app container.

-Sabha

On Wed, Sep 2, 2015 at 4:17 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Thanks for the response. But what I am observing is, by setting
MEMORY_LIMIT as an environment variable does take effect when I restage the
app and it reduces the memory allocated to the java app(not the container
itself) to the amount set in the variable. The app is restarted with the
same heap ratio as dictated by the buildpack but assumes the max memory
available to it is now 1GB instead of 2GB. The container still had 2GB
allocated to it and I understand that will be reduced only if I use the "cf
scale" command.
Is this the expected behavior?

Thanks
Siva

On Wed, Sep 2, 2015 at 3:55 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

This is not a environment variable you can set. To scale down the
memory allocated to your instances, do

cf scale <app-name> -m 1024M

This will tell Warden/Garden to allocate that much memory to your
containers and kill them if they exceed it. The documentation is saying
that the processes running inside the container can learn about the memory
limit applied to them via the environment variable MEMORY_LIMIT. If you
try to set this variable via set-env, it will get overwritten by the value
set by "cf scale". Again, the MEMORY_LIMIT environment variable doesn't do
anything, it's just information passed into the container via the process's
environment. To actually constrain the memory, use "cf scale" or set it in
your manifest file:


https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#memory

On Wed, Sep 2, 2015 at 3:08 PM, Siva Balan <mailsiva(a)gmail.com>
wrote:

We are not using Diego and we are on CF version 211.
I had my application started with 2GB of memory set in the manifest
file. I then ran the following command and restaged the app:
cf set-env <app-name> MEMORY_LIMIT 1024m
Then when I ran "cf app <app-name>", it was still showing the
instance memory of 2GB but the heap size allocated to the application was
cut in half and I confirmed this through New Relic app. Also, the first
full GC of the app occurred when the memory of the instance was at 1.3GB.
So I am not quite sure of the description of the MEMORY_LIMIT env variable
as described in
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#memory.
I did not see my application restart when the instance memory went over
1GB.
Any ideas about this behavior?

Thanks.
Siva

On Tue, Sep 1, 2015 at 7:40 PM, Amit Gupta <agupta(a)pivotal.io>
wrote:

Responses inline.

On Tue, Sep 1, 2015 at 7:13 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

What is the default memory size of an app deployed in diego? cf
apps indicate 1GB.
Yes. This isn't a diego thing, this is a CC thing.


cf set-env doesn't appear to be getting the environment variable
information to the running container. Is cf set-env the correct method to
use?
You likely need to restage the application for ENV var changes to
take effect. Unless I'm mistaken, this too is not diego-specific, and
applies to DEA containers as well.


Thanks!
Tom
Best,
Amit

--
http://www.twitter.com/sivabalans

--
http://www.twitter.com/sivabalans
--
Pivotal CF Architect
rkelapure(a)pivotal.io
cell: 9197242524


Re: diego questions: default memory and passing env variables

Siva Balan <mailsiva@...>
 

Exactly. That is why we are not pursuing changing the BP option. Thanks for
the confirmation.

On Wed, Sep 2, 2015 at 8:47 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Tweaking the buildpack is less dynamic, and every time you want to make a
change you will have to tweak again, create the new buildpack, and upload
it to the CC. Moreover, maintaining your own fork of the buildpack adds
overhead when you need to consume important changes from the standard
buildpack, such as security updates.

On Wed, Sep 2, 2015 at 8:34 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Thanks for the clarification Sabha and Amit. We are using JavaBP and our
usecase is to have additional native memory headroom for the app. One
option is to tweak the open_jdk.yml file in the JavaBP to give more native
memory and reduce the heap in that process or to set this MEMORY_LIMIT env
variable. We are going with setting the MEMORY_LIMIT option.

Regards,
Siva

On Wed, Sep 2, 2015 at 5:35 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Ah, I was mistaken. If you set MEMORY_LIMIT, it will not get
overwritten. So if you don't set it, it will be set to the container
memory limit. But if you do set it, then the java buildpack can take
advantage of it in the way Sabha explained.

On Wed, Sep 2, 2015 at 5:18 PM, Sabha Parameswaran <sabhap(a)pivotal.io>
wrote:

The MEMORY_LIMIT is used by the java buildpack to configure the various
heap parameters for a java app. This variable value is set to the container
size by default by the Cloud controller when you push the app and JBP uses
that to arrive at heap during staging.

What you have done is making the buildpack work with a lower memory
threshold even though the app itself is allowed a higher limit.

Tweak the MEMORY_LIMIT variable only when you are using JavaBP and want
to really have some additional native memory headroom to grow (due to
native memory usage or safety concerns) or to limit heap from utilizing the
full container memory size.

As Amit mentioned, always use cf scale option to change the memory or
instance count for the app container.

-Sabha

On Wed, Sep 2, 2015 at 4:17 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Thanks for the response. But what I am observing is, by setting
MEMORY_LIMIT as an environment variable does take effect when I restage the
app and it reduces the memory allocated to the java app(not the container
itself) to the amount set in the variable. The app is restarted with the
same heap ratio as dictated by the buildpack but assumes the max memory
available to it is now 1GB instead of 2GB. The container still had 2GB
allocated to it and I understand that will be reduced only if I use the "cf
scale" command.
Is this the expected behavior?

Thanks
Siva

On Wed, Sep 2, 2015 at 3:55 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

This is not a environment variable you can set. To scale down the
memory allocated to your instances, do

cf scale <app-name> -m 1024M

This will tell Warden/Garden to allocate that much memory to your
containers and kill them if they exceed it. The documentation is saying
that the processes running inside the container can learn about the memory
limit applied to them via the environment variable MEMORY_LIMIT. If you
try to set this variable via set-env, it will get overwritten by the value
set by "cf scale". Again, the MEMORY_LIMIT environment variable doesn't do
anything, it's just information passed into the container via the process's
environment. To actually constrain the memory, use "cf scale" or set it in
your manifest file:


https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#memory

On Wed, Sep 2, 2015 at 3:08 PM, Siva Balan <mailsiva(a)gmail.com>
wrote:

We are not using Diego and we are on CF version 211.
I had my application started with 2GB of memory set in the manifest
file. I then ran the following command and restaged the app:
cf set-env <app-name> MEMORY_LIMIT 1024m
Then when I ran "cf app <app-name>", it was still showing the
instance memory of 2GB but the heap size allocated to the application was
cut in half and I confirmed this through New Relic app. Also, the first
full GC of the app occurred when the memory of the instance was at 1.3GB.
So I am not quite sure of the description of the MEMORY_LIMIT env variable
as described in
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#memory.
I did not see my application restart when the instance memory went over
1GB.
Any ideas about this behavior?

Thanks.
Siva

On Tue, Sep 1, 2015 at 7:40 PM, Amit Gupta <agupta(a)pivotal.io>
wrote:

Responses inline.

On Tue, Sep 1, 2015 at 7:13 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

What is the default memory size of an app deployed in diego? cf
apps indicate 1GB.
Yes. This isn't a diego thing, this is a CC thing.


cf set-env doesn't appear to be getting the environment variable
information to the running container. Is cf set-env the correct method to
use?
You likely need to restage the application for ENV var changes to
take effect. Unless I'm mistaken, this too is not diego-specific, and
applies to DEA containers as well.


Thanks!
Tom
Best,
Amit

--
http://www.twitter.com/sivabalans

--
http://www.twitter.com/sivabalans


Re: diego questions: default memory and passing env variables

Amit Kumar Gupta
 

Tweaking the buildpack is less dynamic, and every time you want to make a
change you will have to tweak again, create the new buildpack, and upload
it to the CC. Moreover, maintaining your own fork of the buildpack adds
overhead when you need to consume important changes from the standard
buildpack, such as security updates.

On Wed, Sep 2, 2015 at 8:34 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Thanks for the clarification Sabha and Amit. We are using JavaBP and our
usecase is to have additional native memory headroom for the app. One
option is to tweak the open_jdk.yml file in the JavaBP to give more native
memory and reduce the heap in that process or to set this MEMORY_LIMIT env
variable. We are going with setting the MEMORY_LIMIT option.

Regards,
Siva

On Wed, Sep 2, 2015 at 5:35 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Ah, I was mistaken. If you set MEMORY_LIMIT, it will not get
overwritten. So if you don't set it, it will be set to the container
memory limit. But if you do set it, then the java buildpack can take
advantage of it in the way Sabha explained.

On Wed, Sep 2, 2015 at 5:18 PM, Sabha Parameswaran <sabhap(a)pivotal.io>
wrote:

The MEMORY_LIMIT is used by the java buildpack to configure the various
heap parameters for a java app. This variable value is set to the container
size by default by the Cloud controller when you push the app and JBP uses
that to arrive at heap during staging.

What you have done is making the buildpack work with a lower memory
threshold even though the app itself is allowed a higher limit.

Tweak the MEMORY_LIMIT variable only when you are using JavaBP and want
to really have some additional native memory headroom to grow (due to
native memory usage or safety concerns) or to limit heap from utilizing the
full container memory size.

As Amit mentioned, always use cf scale option to change the memory or
instance count for the app container.

-Sabha

On Wed, Sep 2, 2015 at 4:17 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Thanks for the response. But what I am observing is, by setting
MEMORY_LIMIT as an environment variable does take effect when I restage the
app and it reduces the memory allocated to the java app(not the container
itself) to the amount set in the variable. The app is restarted with the
same heap ratio as dictated by the buildpack but assumes the max memory
available to it is now 1GB instead of 2GB. The container still had 2GB
allocated to it and I understand that will be reduced only if I use the "cf
scale" command.
Is this the expected behavior?

Thanks
Siva

On Wed, Sep 2, 2015 at 3:55 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

This is not a environment variable you can set. To scale down the
memory allocated to your instances, do

cf scale <app-name> -m 1024M

This will tell Warden/Garden to allocate that much memory to your
containers and kill them if they exceed it. The documentation is saying
that the processes running inside the container can learn about the memory
limit applied to them via the environment variable MEMORY_LIMIT. If you
try to set this variable via set-env, it will get overwritten by the value
set by "cf scale". Again, the MEMORY_LIMIT environment variable doesn't do
anything, it's just information passed into the container via the process's
environment. To actually constrain the memory, use "cf scale" or set it in
your manifest file:

https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#memory

On Wed, Sep 2, 2015 at 3:08 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

We are not using Diego and we are on CF version 211.
I had my application started with 2GB of memory set in the manifest
file. I then ran the following command and restaged the app:
cf set-env <app-name> MEMORY_LIMIT 1024m
Then when I ran "cf app <app-name>", it was still showing the
instance memory of 2GB but the heap size allocated to the application was
cut in half and I confirmed this through New Relic app. Also, the first
full GC of the app occurred when the memory of the instance was at 1.3GB.
So I am not quite sure of the description of the MEMORY_LIMIT env variable
as described in
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#memory.
I did not see my application restart when the instance memory went over
1GB.
Any ideas about this behavior?

Thanks.
Siva

On Tue, Sep 1, 2015 at 7:40 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Responses inline.

On Tue, Sep 1, 2015 at 7:13 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

What is the default memory size of an app deployed in diego? cf
apps indicate 1GB.
Yes. This isn't a diego thing, this is a CC thing.


cf set-env doesn't appear to be getting the environment variable
information to the running container. Is cf set-env the correct method to
use?
You likely need to restage the application for ENV var changes to
take effect. Unless I'm mistaken, this too is not diego-specific, and
applies to DEA containers as well.


Thanks!
Tom
Best,
Amit

--
http://www.twitter.com/sivabalans

--
http://www.twitter.com/sivabalans


Re: diego questions: default memory and passing env variables

Siva Balan <mailsiva@...>
 

Thanks for the clarification Sabha and Amit. We are using JavaBP and our
usecase is to have additional native memory headroom for the app. One
option is to tweak the open_jdk.yml file in the JavaBP to give more native
memory and reduce the heap in that process or to set this MEMORY_LIMIT env
variable. We are going with setting the MEMORY_LIMIT option.

Regards,
Siva

On Wed, Sep 2, 2015 at 5:35 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Ah, I was mistaken. If you set MEMORY_LIMIT, it will not get
overwritten. So if you don't set it, it will be set to the container
memory limit. But if you do set it, then the java buildpack can take
advantage of it in the way Sabha explained.

On Wed, Sep 2, 2015 at 5:18 PM, Sabha Parameswaran <sabhap(a)pivotal.io>
wrote:

The MEMORY_LIMIT is used by the java buildpack to configure the various
heap parameters for a java app. This variable value is set to the container
size by default by the Cloud controller when you push the app and JBP uses
that to arrive at heap during staging.

What you have done is making the buildpack work with a lower memory
threshold even though the app itself is allowed a higher limit.

Tweak the MEMORY_LIMIT variable only when you are using JavaBP and want
to really have some additional native memory headroom to grow (due to
native memory usage or safety concerns) or to limit heap from utilizing the
full container memory size.

As Amit mentioned, always use cf scale option to change the memory or
instance count for the app container.

-Sabha

On Wed, Sep 2, 2015 at 4:17 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Thanks for the response. But what I am observing is, by setting
MEMORY_LIMIT as an environment variable does take effect when I restage the
app and it reduces the memory allocated to the java app(not the container
itself) to the amount set in the variable. The app is restarted with the
same heap ratio as dictated by the buildpack but assumes the max memory
available to it is now 1GB instead of 2GB. The container still had 2GB
allocated to it and I understand that will be reduced only if I use the "cf
scale" command.
Is this the expected behavior?

Thanks
Siva

On Wed, Sep 2, 2015 at 3:55 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

This is not a environment variable you can set. To scale down the
memory allocated to your instances, do

cf scale <app-name> -m 1024M

This will tell Warden/Garden to allocate that much memory to your
containers and kill them if they exceed it. The documentation is saying
that the processes running inside the container can learn about the memory
limit applied to them via the environment variable MEMORY_LIMIT. If you
try to set this variable via set-env, it will get overwritten by the value
set by "cf scale". Again, the MEMORY_LIMIT environment variable doesn't do
anything, it's just information passed into the container via the process's
environment. To actually constrain the memory, use "cf scale" or set it in
your manifest file:

https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#memory

On Wed, Sep 2, 2015 at 3:08 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

We are not using Diego and we are on CF version 211.
I had my application started with 2GB of memory set in the manifest
file. I then ran the following command and restaged the app:
cf set-env <app-name> MEMORY_LIMIT 1024m
Then when I ran "cf app <app-name>", it was still showing the instance
memory of 2GB but the heap size allocated to the application was cut in
half and I confirmed this through New Relic app. Also, the first full GC of
the app occurred when the memory of the instance was at 1.3GB. So I am not
quite sure of the description of the MEMORY_LIMIT env variable as described
in
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#memory.
I did not see my application restart when the instance memory went over
1GB.
Any ideas about this behavior?

Thanks.
Siva

On Tue, Sep 1, 2015 at 7:40 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Responses inline.

On Tue, Sep 1, 2015 at 7:13 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

What is the default memory size of an app deployed in diego? cf apps
indicate 1GB.
Yes. This isn't a diego thing, this is a CC thing.


cf set-env doesn't appear to be getting the environment variable
information to the running container. Is cf set-env the correct method to
use?
You likely need to restage the application for ENV var changes to
take effect. Unless I'm mistaken, this too is not diego-specific, and
applies to DEA containers as well.


Thanks!
Tom
Best,
Amit

--
http://www.twitter.com/sivabalans


CAB September Call on 9/9/2015 @ 8a PDT

Michael Maximilien
 

Hi, all,

Quick reminder that the CAB call for September is next week Wednesday
September 9th @ 8a PDT.

Please add any project updates to Agenda here:
https://docs.google.com/document/d/1SCOlAquyUmNM-AQnekCOXiwhLs6gveTxAcduvDcW_xI/edit#heading=h.o44xhgvum2we

If you have something else to share, please also add an entry at the end.

Best,

Chip, James, and Max

PS: Dr.Nic this is one week in advance, so no excuses ;) phone info listed
on agenda.
PPS: Have a great labor day weekend---if you are in the US.


Re: diego questions: default memory and passing env variables

Amit Kumar Gupta
 

Ah, I was mistaken. If you set MEMORY_LIMIT, it will not get overwritten.
So if you don't set it, it will be set to the container memory limit. But
if you do set it, then the java buildpack can take advantage of it in the
way Sabha explained.

On Wed, Sep 2, 2015 at 5:18 PM, Sabha Parameswaran <sabhap(a)pivotal.io>
wrote:

The MEMORY_LIMIT is used by the java buildpack to configure the various
heap parameters for a java app. This variable value is set to the container
size by default by the Cloud controller when you push the app and JBP uses
that to arrive at heap during staging.

What you have done is making the buildpack work with a lower memory
threshold even though the app itself is allowed a higher limit.

Tweak the MEMORY_LIMIT variable only when you are using JavaBP and want to
really have some additional native memory headroom to grow (due to native
memory usage or safety concerns) or to limit heap from utilizing the full
container memory size.

As Amit mentioned, always use cf scale option to change the memory or
instance count for the app container.

-Sabha

On Wed, Sep 2, 2015 at 4:17 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Thanks for the response. But what I am observing is, by setting
MEMORY_LIMIT as an environment variable does take effect when I restage the
app and it reduces the memory allocated to the java app(not the container
itself) to the amount set in the variable. The app is restarted with the
same heap ratio as dictated by the buildpack but assumes the max memory
available to it is now 1GB instead of 2GB. The container still had 2GB
allocated to it and I understand that will be reduced only if I use the "cf
scale" command.
Is this the expected behavior?

Thanks
Siva

On Wed, Sep 2, 2015 at 3:55 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

This is not a environment variable you can set. To scale down the
memory allocated to your instances, do

cf scale <app-name> -m 1024M

This will tell Warden/Garden to allocate that much memory to your
containers and kill them if they exceed it. The documentation is saying
that the processes running inside the container can learn about the memory
limit applied to them via the environment variable MEMORY_LIMIT. If you
try to set this variable via set-env, it will get overwritten by the value
set by "cf scale". Again, the MEMORY_LIMIT environment variable doesn't do
anything, it's just information passed into the container via the process's
environment. To actually constrain the memory, use "cf scale" or set it in
your manifest file:

https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#memory

On Wed, Sep 2, 2015 at 3:08 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

We are not using Diego and we are on CF version 211.
I had my application started with 2GB of memory set in the manifest
file. I then ran the following command and restaged the app:
cf set-env <app-name> MEMORY_LIMIT 1024m
Then when I ran "cf app <app-name>", it was still showing the instance
memory of 2GB but the heap size allocated to the application was cut in
half and I confirmed this through New Relic app. Also, the first full GC of
the app occurred when the memory of the instance was at 1.3GB. So I am not
quite sure of the description of the MEMORY_LIMIT env variable as described
in
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#memory.
I did not see my application restart when the instance memory went over
1GB.
Any ideas about this behavior?

Thanks.
Siva

On Tue, Sep 1, 2015 at 7:40 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Responses inline.

On Tue, Sep 1, 2015 at 7:13 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

What is the default memory size of an app deployed in diego? cf apps
indicate 1GB.
Yes. This isn't a diego thing, this is a CC thing.


cf set-env doesn't appear to be getting the environment variable
information to the running container. Is cf set-env the correct method to
use?
You likely need to restage the application for ENV var changes to take
effect. Unless I'm mistaken, this too is not diego-specific, and applies
to DEA containers as well.


Thanks!
Tom
Best,
Amit

--
http://www.twitter.com/sivabalans


Re: diego questions: default memory and passing env variables

Sabha
 

The MEMORY_LIMIT is used by the java buildpack to configure the various
heap parameters for a java app. This variable value is set to the container
size by default by the Cloud controller when you push the app and JBP uses
that to arrive at heap during staging.

What you have done is making the buildpack work with a lower memory
threshold even though the app itself is allowed a higher limit.

Tweak the MEMORY_LIMIT variable only when you are using JavaBP and want to
really have some additional native memory headroom to grow (due to native
memory usage or safety concerns) or to limit heap from utilizing the full
container memory size.

As Amit mentioned, always use cf scale option to change the memory or
instance count for the app container.

-Sabha

On Wed, Sep 2, 2015 at 4:17 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

Thanks for the response. But what I am observing is, by setting
MEMORY_LIMIT as an environment variable does take effect when I restage the
app and it reduces the memory allocated to the java app(not the container
itself) to the amount set in the variable. The app is restarted with the
same heap ratio as dictated by the buildpack but assumes the max memory
available to it is now 1GB instead of 2GB. The container still had 2GB
allocated to it and I understand that will be reduced only if I use the "cf
scale" command.
Is this the expected behavior?

Thanks
Siva

On Wed, Sep 2, 2015 at 3:55 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

This is not a environment variable you can set. To scale down the memory
allocated to your instances, do

cf scale <app-name> -m 1024M

This will tell Warden/Garden to allocate that much memory to your
containers and kill them if they exceed it. The documentation is saying
that the processes running inside the container can learn about the memory
limit applied to them via the environment variable MEMORY_LIMIT. If you
try to set this variable via set-env, it will get overwritten by the value
set by "cf scale". Again, the MEMORY_LIMIT environment variable doesn't do
anything, it's just information passed into the container via the process's
environment. To actually constrain the memory, use "cf scale" or set it in
your manifest file:

https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#memory

On Wed, Sep 2, 2015 at 3:08 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

We are not using Diego and we are on CF version 211.
I had my application started with 2GB of memory set in the manifest
file. I then ran the following command and restaged the app:
cf set-env <app-name> MEMORY_LIMIT 1024m
Then when I ran "cf app <app-name>", it was still showing the instance
memory of 2GB but the heap size allocated to the application was cut in
half and I confirmed this through New Relic app. Also, the first full GC of
the app occurred when the memory of the instance was at 1.3GB. So I am not
quite sure of the description of the MEMORY_LIMIT env variable as described
in
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#memory.
I did not see my application restart when the instance memory went over
1GB.
Any ideas about this behavior?

Thanks.
Siva

On Tue, Sep 1, 2015 at 7:40 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Responses inline.

On Tue, Sep 1, 2015 at 7:13 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

What is the default memory size of an app deployed in diego? cf apps
indicate 1GB.
Yes. This isn't a diego thing, this is a CC thing.


cf set-env doesn't appear to be getting the environment variable
information to the running container. Is cf set-env the correct method to
use?
You likely need to restage the application for ENV var changes to take
effect. Unless I'm mistaken, this too is not diego-specific, and applies
to DEA containers as well.


Thanks!
Tom
Best,
Amit

--
http://www.twitter.com/sivabalans


Re: diego questions: default memory and passing env variables

Siva Balan <mailsiva@...>
 

Thanks for the response. But what I am observing is, by setting
MEMORY_LIMIT as an environment variable does take effect when I restage the
app and it reduces the memory allocated to the java app(not the container
itself) to the amount set in the variable. The app is restarted with the
same heap ratio as dictated by the buildpack but assumes the max memory
available to it is now 1GB instead of 2GB. The container still had 2GB
allocated to it and I understand that will be reduced only if I use the "cf
scale" command.
Is this the expected behavior?

Thanks
Siva

On Wed, Sep 2, 2015 at 3:55 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

This is not a environment variable you can set. To scale down the memory
allocated to your instances, do

cf scale <app-name> -m 1024M

This will tell Warden/Garden to allocate that much memory to your
containers and kill them if they exceed it. The documentation is saying
that the processes running inside the container can learn about the memory
limit applied to them via the environment variable MEMORY_LIMIT. If you
try to set this variable via set-env, it will get overwritten by the value
set by "cf scale". Again, the MEMORY_LIMIT environment variable doesn't do
anything, it's just information passed into the container via the process's
environment. To actually constrain the memory, use "cf scale" or set it in
your manifest file:

https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#memory

On Wed, Sep 2, 2015 at 3:08 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

We are not using Diego and we are on CF version 211.
I had my application started with 2GB of memory set in the manifest file.
I then ran the following command and restaged the app:
cf set-env <app-name> MEMORY_LIMIT 1024m
Then when I ran "cf app <app-name>", it was still showing the instance
memory of 2GB but the heap size allocated to the application was cut in
half and I confirmed this through New Relic app. Also, the first full GC of
the app occurred when the memory of the instance was at 1.3GB. So I am not
quite sure of the description of the MEMORY_LIMIT env variable as described
in
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#memory.
I did not see my application restart when the instance memory went over
1GB.
Any ideas about this behavior?

Thanks.
Siva

On Tue, Sep 1, 2015 at 7:40 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Responses inline.

On Tue, Sep 1, 2015 at 7:13 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

What is the default memory size of an app deployed in diego? cf apps
indicate 1GB.
Yes. This isn't a diego thing, this is a CC thing.


cf set-env doesn't appear to be getting the environment variable
information to the running container. Is cf set-env the correct method to
use?
You likely need to restage the application for ENV var changes to take
effect. Unless I'm mistaken, this too is not diego-specific, and applies
to DEA containers as well.


Thanks!
Tom
Best,
Amit


Re: diego questions: default memory and passing env variables

Amit Kumar Gupta
 

This is not a environment variable you can set. To scale down the memory
allocated to your instances, do

cf scale <app-name> -m 1024M

This will tell Warden/Garden to allocate that much memory to your
containers and kill them if they exceed it. The documentation is saying
that the processes running inside the container can learn about the memory
limit applied to them via the environment variable MEMORY_LIMIT. If you
try to set this variable via set-env, it will get overwritten by the value
set by "cf scale". Again, the MEMORY_LIMIT environment variable doesn't do
anything, it's just information passed into the container via the process's
environment. To actually constrain the memory, use "cf scale" or set it in
your manifest file:

https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.html#memory

On Wed, Sep 2, 2015 at 3:08 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

We are not using Diego and we are on CF version 211.
I had my application started with 2GB of memory set in the manifest file.
I then ran the following command and restaged the app:
cf set-env <app-name> MEMORY_LIMIT 1024m
Then when I ran "cf app <app-name>", it was still showing the instance
memory of 2GB but the heap size allocated to the application was cut in
half and I confirmed this through New Relic app. Also, the first full GC of
the app occurred when the memory of the instance was at 1.3GB. So I am not
quite sure of the description of the MEMORY_LIMIT env variable as described
in
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#memory.
I did not see my application restart when the instance memory went over
1GB.
Any ideas about this behavior?

Thanks.
Siva

On Tue, Sep 1, 2015 at 7:40 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Responses inline.

On Tue, Sep 1, 2015 at 7:13 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

What is the default memory size of an app deployed in diego? cf apps
indicate 1GB.
Yes. This isn't a diego thing, this is a CC thing.


cf set-env doesn't appear to be getting the environment variable
information to the running container. Is cf set-env the correct method to
use?
You likely need to restage the application for ENV var changes to take
effect. Unless I'm mistaken, this too is not diego-specific, and applies
to DEA containers as well.


Thanks!
Tom
Best,
Amit


Re: diego questions: default memory and passing env variables

Matthew Sykes <matthew.sykes@...>
 

Memory limits are set via cf scale.

`cf scale <app-name> -m 512M`

will scale your app to 512MB per instance. The cli will then tell you to
restage your application so the buildpacks can observe the new limit and
make the necessary changes. At one time, this was especially true of the
java buildpack.

On Wed, Sep 2, 2015 at 6:08 PM, Siva Balan <mailsiva(a)gmail.com> wrote:

We are not using Diego and we are on CF version 211.
I had my application started with 2GB of memory set in the manifest file.
I then ran the following command and restaged the app:
cf set-env <app-name> MEMORY_LIMIT 1024m
Then when I ran "cf app <app-name>", it was still showing the instance
memory of 2GB but the heap size allocated to the application was cut in
half and I confirmed this through New Relic app. Also, the first full GC of
the app occurred when the memory of the instance was at 1.3GB. So I am not
quite sure of the description of the MEMORY_LIMIT env variable as described
in
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#memory.
I did not see my application restart when the instance memory went over
1GB.
Any ideas about this behavior?

Thanks.
Siva

On Tue, Sep 1, 2015 at 7:40 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Responses inline.

On Tue, Sep 1, 2015 at 7:13 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

What is the default memory size of an app deployed in diego? cf apps
indicate 1GB.
Yes. This isn't a diego thing, this is a CC thing.


cf set-env doesn't appear to be getting the environment variable
information to the running container. Is cf set-env the correct method to
use?
You likely need to restage the application for ENV var changes to take
effect. Unless I'm mistaken, this too is not diego-specific, and applies
to DEA containers as well.


Thanks!
Tom
Best,
Amit
--
Matthew Sykes
matthew.sykes(a)gmail.com


Re: diego questions: default memory and passing env variables

Siva Balan <mailsiva@...>
 

We are not using Diego and we are on CF version 211.
I had my application started with 2GB of memory set in the manifest file. I
then ran the following command and restaged the app:
cf set-env <app-name> MEMORY_LIMIT 1024m
Then when I ran "cf app <app-name>", it was still showing the instance
memory of 2GB but the heap size allocated to the application was cut in
half and I confirmed this through New Relic app. Also, the first full GC of
the app occurred when the memory of the instance was at 1.3GB. So I am not
quite sure of the description of the MEMORY_LIMIT env variable as described
in
http://docs.run.pivotal.io/devguide/deploy-apps/environment-variable.html#memory.
I did not see my application restart when the instance memory went over
1GB.
Any ideas about this behavior?

Thanks.
Siva

On Tue, Sep 1, 2015 at 7:40 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Responses inline.

On Tue, Sep 1, 2015 at 7:13 PM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:

What is the default memory size of an app deployed in diego? cf apps
indicate 1GB.
Yes. This isn't a diego thing, this is a CC thing.


cf set-env doesn't appear to be getting the environment variable
information to the running container. Is cf set-env the correct method to
use?
You likely need to restage the application for ENV var changes to take
effect. Unless I'm mistaken, this too is not diego-specific, and applies
to DEA containers as well.


Thanks!
Tom
Best,
Amit


Re: Generic data points for dropsonde

Johannes Tuchscherer
 

The current way of sending metrics as either Values or Counters through the
pipeline makes the development of a downstream consumer (=nozzle) pretty
easy. If you look at the datadog nozzle[0], it just takes all ValueMetrics
and Counters and sends them off to datadog. The nozzle does not have to
know anything about these metrics (e.g. their origin, name, or layout).

Adding a new way to send metrics as a nested object would make the
downstream implementation certainly more complicated. In that case, the
nozzle developer has to know what metrics are included inside the generic
point (basically the schema of the metric) and treat each point
accordingly. For example, if I were to write a nozzle that emits metrics to
Graphite with a StatsD client (like it is done here[1]), I need to know if
my int64 value is a Gauge or a Counter. Also, my consumer is under constant
risk of breaking when the upstream schema changes.

We are already facing this problem with the container metrics. But at least
the container metrics are in a defined format that is well documented and
not likely to change.

I agree with you, though, the the dropsonde protocol could use a mechanism
for easier extension. Having a GenericPoint and/or GenericEvent seems like
a good idea that I whole-heartedly support. I would just like to stay away
from nested metrics. I think the cost of adding more logic into the
downstream consumer (and making it harder to maintain) is not worth the
benefit of a more concise metric transport.


[0] https://github.com/cloudfoundry-incubator/datadog-firehose-nozzle
[1] https://github.com/CloudCredo/graphite-nozzle

On Tue, Sep 1, 2015 at 5:52 PM, Benjamin Black <bblack(a)pivotal.io> wrote:

great questions, dwayne.

1) the partition key is intended to be used in a similar manner to
partitioners in distributed systems like cassandra or kafka. the specific
behavior i would like to make part of the contract is two-fold: that all
data with the same key is routed to the same partition and that all data in
a partition is FIFO (meaning no ordering guarantees beyond arrival time).

this could help with the multi-line log problem by ensuring a single
consumer will receive all lines for a given log entry in order, allowing
simple reassembly. however, the lines might be interleaved with other lines
with the same key or even other keys that happen to map to the same
partition, so the consumer does require a bit of intelligence. this was
actually one of the driving scenarios for adding the key.

2) i expect typical points to be in the hundreds of bytes to a few KB. if
we find ourselves regularly needing much larger points, especially near
that 64KB limit, i'd look to the JSON representation as the hierarchical
structure is more efficiently managed there.


b




On Tue, Sep 1, 2015 at 4:42 PM, <dschultz(a)pivotal.io> wrote:

Hi Ben,

I was wondering if you could give a concrete use case for the partition
key functionality.

In particular I am interested in how we solve multi line log entries. I
think it would be better to solve it by keeping all the data (the multiple
lines) together throughout the logging/metrics pipeline, but could see how
something like a partition key might help keep the data together as well.

Second question: how large do you see these point messages getting
(average and max)? There are still several stages of the logging/metrics
pipeline that use UDP with a standard 64K size limit.

Thanks,
Dwayne

On Aug 28, 2015, at 4:54 PM, Benjamin Black <bblack(a)pivotal.io> wrote:

All,

The existing dropsonde protocol uses a different message type for each
event type. HttpStart, HttpStop, ContainerMetrics, and so on are all
distinct types in the protocol definition. This requires protocol changes
to introduce any new event type, making such changes very expensive. We've
been working for the past few weeks on an addition to the dropsonde
protocol to support easier future extension to new types of events and to
make it easier for users to define their own events.

The document linked below [1] describes a generic data point message
capable of carrying multi-dimensional, multi-metric points as sets of
name/value pairs. This new message is expected to be added as an additional
entry in the existing dropsonde protocol metric type enum. Things are now
at a point where we'd like to get feedback from the community before moving
forward with implementation.

Please contribute your thoughts on the document in whichever way you are
most comfortable: comments on the document, email here, or email directly
to me. If you comment on the document, please make sure you are logged in
so we can keep track of who is asking for what. Your views are not just
appreciated, but critical to the continued health and success of the Cloud
Foundry community. Thank you!


b

[1]
https://docs.google.com/document/d/1SzvT1BjrBPqUw6zfSYYFfaW9vX_dTZZjn5sl2nxB6Bc/edit?usp=sharing





Re: Correcting CF Docs

Joaquin Zalazar <joaquin.zalazar@...>
 

Hi Daniel!

What I've done while testing is to set the variable BOSH_AWS_REGION to the
region I needed.

export BOSH_AWS_REGION=<your region here>

Regards!!

Joaquin Zalazar

Altoros


On Wed, Sep 2, 2015 at 9:43 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Hi all,

The CF docs for deploying CF to an AWS VPC
<https://docs.cloudfoundry.org/deploying/ec2/bootstrap-aws-vpc.html> have
a broken link to a Gist of Dan Highham's, that presumably talks about
deploying to regions other than us-east-1.

Does anyone know where the instructions for non-US regions are now?

Regards,

Daniel Jones
EngineerBetter.com


--

*Joaquin Zalazar*
*ALTOROS* — Cloud Foundry deployment, training and integration
www.altoros.com


Re: docker-push failed with crashing and got the following stack trace in grarden-linux

Shaozhen Ding
 

Thanks for the help

I had some permission issues on several files. Since diego run docker
containers with user vcap. I created a user named as vcap and gave the file
permission to vcap in advanced. Then i got the error above, even though I
created the home folder.

I remove the user, then the error is gone. Definitely curious of what diego
is trying to do with vcap user.

On Wed, Sep 2, 2015 at 11:40 AM, Will Pragnell <wpragnell(a)pivotal.io> wrote:

The error message suggests that in `/etc/passwd` in the Docker image, the
entry for `vcap` lists a path for the home dir that doesn't exist. I'm
afraid I don't understand Diego well enough to know why it's trying to
stream into the container as `vcap` though.

On 2 September 2015 at 17:25, Shaozhen Ding <dsz0111(a)gmail.com> wrote:

Anyone can help with interpreting the error log here
garden-linux.pool.vl5hraecvg1.stream-in.command.failed

cf-213 and diego/0.1353.0


{"timestamp":"1441209896.800271988","source":"garden-linux","message":"garden-linux.garden-server.create.created","log_level":1,"data":{"request":{"handle":"cb6ac15a-c3a0-4b5d-9716-e99934c35953-2669bf25-c226-4908-bbba-63e00a6a4fa0-397eef75-bf73-46d3-6ba2-6a3737ea3d5e","rootfs":"docker:///datianshi/cf-docker-music#latest","properties":{"executor:allocated-at":"1441209893185279557","executor:cpu-weight":"4","executor:disk-mb":"1024","executor:log-config":"{\"guid\":\"cb6ac15a-c3a0-4b5d-9716-e99934c35953\",\"index\":0,\"source_name\":\"CELL\"}","executor:memory-mb":"512","executor:metrics-config":"{\"guid\":\"cb6ac15a-c3a0-4b5d-9716-e99934c35953\",\"index\":0}","executor:owner":"executor","executor:result":"{\"failed\":false,\"failure_reason\":\"\",\"stopped\":false}","executor:rootfs":"docker:///datianshi/cf-docker-music#latest","executor:start-timeout":"60","executor:state":"created","tag:domain":"cf-apps","tag:instance-guid":"397eef75-bf73-46d3-6ba2-6a3737ea3d5e","tag:lifecyc
le":"lrp

","tag:process-guid":"cb6ac15a-c3a0-4b5d-9716-e99934c35953-2669bf25-c226-4908-bbba-63e00a6a4fa0","tag:process-index":"0"},"env":["INSTANCE_GUID=397eef75-bf73-46d3-6ba2-6a3737ea3d5e","INSTANCE_INDEX=0"]},"session":"4.247804"}}

{"timestamp":"1441209896.811697721","source":"garden-linux","message":"garden-linux.garden-server.net-in.port-mapped","log_level":1,"data":{"container-port":8080,"handle":"cb6ac15a-c3a0-4b5d-9716-e99934c35953-2669bf25-c226-4908-bbba-63e00a6a4fa0-397eef75-bf73-46d3-6ba2-6a3737ea3d5e","host-port":61055,"session":"4.247809"}}

{"timestamp":"1441209896.818696976","source":"garden-linux","message":"garden-linux.garden-server.net-in.port-mapped","log_level":1,"data":{"container-port":2222,"handle":"cb6ac15a-c3a0-4b5d-9716-e99934c35953-2669bf25-c226-4908-bbba-63e00a6a4fa0-397eef75-bf73-46d3-6ba2-6a3737ea3d5e","host-port":61056,"session":"4.247810"}}

{"timestamp":"1441209896.842704535","source":"garden-linux","message":"garden-linux.garden-server.limit-memory.limited","log_level":1,"data":{"handle":"cb6ac15a-c3a0-4b5d-9716-e99934c35953-2669bf25-c226-4908-bbba-63e00a6a4fa0-397eef75-bf73-46d3-6ba2-6a3737ea3d5e","resulting-limits":{"limit_in_bytes":536870912},"session":"4.247818"}}

{"timestamp":"1441209896.843993425","source":"garden-linux","message":"garden-linux.garden-server.limit-disk.limited","log_level":1,"data":{"handle":"cb6ac15a-c3a0-4b5d-9716-e99934c35953-2669bf25-c226-4908-bbba-63e00a6a4fa0-397eef75-bf73-46d3-6ba2-6a3737ea3d5e","resulting-limits":{},"session":"4.247819"}}

{"timestamp":"1441209896.844884396","source":"garden-linux","message":"garden-linux.garden-server.limit-cpu.limited","log_level":1,"data":{"handle":"cb6ac15a-c3a0-4b5d-9716-e99934c35953-2669bf25-c226-4908-bbba-63e00a6a4fa0-397eef75-bf73-46d3-6ba2-6a3737ea3d5e","resulting-limits":{"limit_in_shares":40},"session":"4.247820"}}

{"timestamp":"1441209896.845991373","source":"garden-linux","message":"garden-linux.garden-server.info.got-info","log_level":1,"data":{"handle":"cb6ac15a-c3a0-4b5d-9716-e99934c35953-2669bf25-c226-4908-bbba-63e00a6a4fa0-397eef75-bf73-46d3-6ba2-6a3737ea3d5e","session":"4.247821"}}
{"timestamp":"1441209896.857844830","source":"garden-linux","message":"garden-linux.pool.vl5hraecvg1.stream-in.command.failed","log_level":2,"data":{"argv":["/var/vcap/data/garden-linux/depot/vl5hraecvg1/bin/nstar","11856","vcap","/tmp/lifecycle"],"error":"exit
status 1","exit-status":1,"session":"2.67.7.1","stderr":"chdir to user
home: No such file or directory\n","stdout":"","took":"2.959157ms"}}
{"timestamp":"1441209896.858107328","source":"garden-linux","message":"garden-linux.garden-server.stream-in.failed","log_level":2,"data":{"destination":"/tmp/lifecycle","error":"exit
status
1","handle":"cb6ac15a-c3a0-4b5d-9716-e99934c35953-2669bf25-c226-4908-bbba-63e00a6a4fa0-397eef75-bf73-46d3-6ba2-6a3737ea3d5e","session":"4.247823"}}


Re: v3 cc api style guide feedback requested

Zach Robinson
 

Thanks James, I've just corrected the three issues you've noted so far

7941 - 7960 of 9422