Hello Jon,

Can you please let me know what version of Cloud Foundry you are running? If you could include the output of your "info" endpoint it would be great (http://apidocs.cloudfoundry.org/208/info/get_info.html)

Cheers,
Vlad

Hi Piotr,

The cloudfoundry-community github org is open to anyone in the community
who'd like to publicize their code. I don't believe there are any strict
guidelines around the license that should be applied; and in fact this code
is explicitly **not** owned by the CF Foundation.

Instead, this code is owned by its authors and contributors. I'd suggest
that you open a Github Issue on the repo asking for a license to be
explicilty declared, either in the README or in a separate LICENSE file.
Usually authors are very willing to do so once asked.

Cheers,
-mike

Hi Jon,

Just for context, the VS extension was only this week open-sourced by HP
and put into the CF incubator.

I expect this issue is arising because the VS extension was developed to
run with HP Helion, which has followed its own evolutionary path from a
previous version of CF, and so it has not been tested against CF edge.

As the Foundation and HP work together to incubate this project in earnest,
we'll work through these support issues and get it working with current
versions of CF.

I expect these speed bumps will get addressed quickly, and so thanks in
advance for your patience.

-m

Hi Stephen,

DRYRUN mode does everything the same as normal operation *except* for the
actual deletion, so if Chaos Lemur is reporting unexpected names it's a
problem with some other part of your config, not DRYRUN. From your
environment variables it seems you haven't configured it to access your
underlying infrastructure, either AWS or VSphere. You need to provide
credentials[1] for one of those:

AWS_ACCESSKEYID & AWS_SECRETACCESSKEY, or

VSPHERE_HOST & VSPHERE_PASSWORD & VSPHERE_USERNAME

Give that a go and it should work, but be aware that if DRYRUN is not true
then you will lose VMs!

Cheers, Paul

[1]: https://github.com/pivotal-cf/chaos-lemur#environment-variables



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-Problem-for-Chaos-Lemur-tp90p96.html
Sent from the CF Dev mailing list archive at Nabble.com.

The "norouter", while interesting, isn't the official CF project approach
to HTTP traffic routing. The TCP router is being built to support TCP
routing as a general solution, with the goal of it becoming an official
part of the CF release when ready.

-chip

Chip Childers | Technology Chief of Staff | Cloud Foundry Foundation

On Fri, May 15, 2015 at 8:58 AM, Mohamed, Owais <Owais.Mohamed(a)covisint.com>
wrote:

Hi,

I attended sessions on both TCP Router (Cloud Foundry and IOT protocol
support by Atul Kshirsagar) and NoRouter (Norouter: Running Cloud Foundry
without the Gorouter by Mike Heath).

I just wanted to start a discussion on the pros and cons of each
approach.

As personal opinion I think NoRouter is a simpler approach and can
definitely be made to support IOT protocols. The main drawback I see with
the NoRouter is the danger of the LTM becoming a Monolith.

Any suggestions\ideas?

Regards,
Owais


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

You can use the allow_host_access property in your manifest to enable the
access. It simply suppresses the iptables rule you mention that rejects the
traffic.

https://github.com/cloudfoundry/cf-release/blob/master/jobs/dea_next/spec#L37

Aaron Huber
Intel Corporation



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-Capturing-data-from-DEA-varz-endpoints-tp92p95.html
Sent from the CF Dev mailing list archive at Nabble.com.

As part of our CF monitoring we want to collect data from the DEA varz
endpoints to help us calculate our headroom. In the spirit of eating our
own dogfood we want to deploy this a micro service on CF itself.

The challenge we face is the DEA explicitly denies the containers to talk
to the varz endpoint for the DEA the app runs on. It can happily hit varz
for all other DEAs. The deny is handled by an iptable reject on the dea's
interface.

Looking at my options, I could run the app outside of CF or add an iptable
rule to the dea's. Neither option fills me with joy.

How have others approached the problem of collecting varz data for
analysis? Admin-ui does this but runs on a VM, possibly for this very
reason.

Thanks
John.

Hi,

I attended sessions on both TCP Router (Cloud Foundry and IOT protocol support by Atul Kshirsagar) and NoRouter (Norouter: Running Cloud Foundry without the Gorouter by Mike Heath).

I just wanted to start a discussion on the pros and cons of each approach.

As personal opinion I think NoRouter is a simpler approach and can definitely be made to support IOT protocols. The main drawback I see with the NoRouter is the danger of the LTM becoming a Monolith.

Any suggestions\ideas?

Regards,
Owais

Thanks Chris.
We had given the MALLOC_ARENA_MAX setting a wide birth because Lari had
mentioned it was buggy on lucid64 & we're still on that stemcell right now.

We do have a test case running all weekend on Tomcat 8.0.22, which I'll try
to remember to post back to the list on.





--
View this message in context: http://cf-dev.70369.x6.nabble.com/Re-cf-dev-Cf-dev-vcap-dev-Java-OOM-debugging-tp25p91.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi guys,

I am trying to use the tool Chaos Lemur to do HA test for CF. I got the guide from below link: http://blog.pivotal.io/cloud-foundry-pivotal/products/chaos-lemur-testing-high-availability-on-pivotal-cloud-foundry

My problem is:
Chaos Lemur seems always running in ‘DRYRUN’ mode(which means it does not destroy VMs actually). Below is the log(get from command: cf logs chaos-lemur):

2015-05-13T14:40:00.00+0800 [App/0] OUT [CHAOS LEMUR] INFO f3a09be7-8bc4-4bed-94dc-32f1a3a653f6 Beginning run...
2015-05-13T14:40:00.00+0800 [App/0] OUT [CHAOS LEMUR] INFO f3a09be7-8bc4-4bed-94dc-32f1a3a653f6 Destroyed: [id: id-3, deployment: deployment-3, job: job-3, name: name-3]
2015-05-13T14:40:00.00+0800 [App/0] OUT [CHAOS LEMUR] INFO f3a09be7-8bc4-4bed-94dc-32f1a3a653f6 Destroyed: [id: id-2, deployment: deployment-2, job: job-2, name: name-2]
2015-05-13T14:40:00.00+0800 [App/0] OUT [CHAOS LEMUR] INFO Chaos Lemur Destruction (f3a09be7-8bc4-4bed-94dc-32f1a3a653f6):
2015-05-13T14:40:00.00+0800 [App/0] OUT 2 VMs destroyed:
2015-05-13T14:40:00.00+0800 [App/0] OUT * name-2
2015-05-13T14:40:00.00+0800 [App/0] OUT * name-3

It seems that the tool is running in ‘DRYRUN’ mode because the destroyed ‘id’, ‘deployment’ and ‘job’ seems not be real names. My question is: I already set the environment variable ‘DRYRUN’ as false(also tried not set this variable). Do you have any idea?

Below are my environment variable setting:
User-Provided:
DIRECTOR_HOST: x.x.x.x
DIRECTOR_PASSWORD: admin
DIRECTOR_USERNAME: admin
DRYRUN: false
SCHEDULE: 0 0/5 * * * *
swift-dev_PROBABILITY: 0.5
swift_storage_PROBABILITY: 0.5

‘swift-dev’ is a deployment name and ‘swift_storage’ is a job name under previous deployment. I got the deployment and job from command: bosh vms. It will list all the deployments and their jobs. Is there any problems with my settings? I think if it is because that I do not get the correct ‘deployment’ and ‘job’?



Any help will be highly appreciated!



Stephen Zhou (周红星)
Mobile: +86 13683173521
Email: stephen.zhou(a)emc.com<mailto:Xu_Louie(a)emc.com>

Hey Guangcai:

Dr. Nic Williams from Stark & Wayne blogged [0] a pretty nice write up
about billing. He did a nice job updating it with the new endpoints that
Dieu noted earlier.

[0]
https://blog.starkandwayne.com/2015/01/22/billing-your-cloud-foundry-users/

Greg Oehmen
Cloud Foundry Product Manager
415.205.6596

Hi Guancai,

This manifest property and api's have been deprecated as they calculated
things incorrectly.
They have been replaced with App Usage Events and Service Usage Events
which you can find more info on in the apidocs. [1]

-Dieu Cao
CF Runtime PM

[1]

On Thu, May 14, 2015 at 5:01 PM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

Hi,

I am trying to understand the property "cc:billing_event_writing_enabled".
Who can share some knowledge on my following questions? if I enable this
property, when will the billing events be written? How long will the
billing events be kept?

Thanks

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

On 15-05-14 10:23 AM, Daniel Jones wrote:

Do you know how Tomcat's APR/NIO memory gets allocated? Is there a way
of telling from pmap whether pages are being used for NIO buffers or
by the APR?

The JVM has Native Memory Tracking (NMT), but I've never used it. Here
are some details about it:
http://hirt.se/blog/?p=401
http://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html
http://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html#BABIIIAC
By using NMT, you can find out what memory regions are used by the JVM.
That would help finding out what areas of the pmap report aren't tracked
by NMT and NMT reports will give insight what the tracked memory is used
for.

I assume that won't track any allocations made in custom native
libraries like the Tomcat native library.
The native memory used by zip handling in the JVM doesn't seem to be
tracked either:
http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/a006fa0a9e8f/src/share/native/java/util/zip/zip_util.c#l627
The "-Dsun.zip.disableMemoryMapping=true" setting changes how memory
allocation is done in the zip functions of the JDK. It doesn't get rid
of native memory allocations, but just makes it not use mmap. I'm not
sure if that leads to anything useful, but it's worth trying if you have
a good test environment.

Lari

Hi,

I am trying to understand the property "cc:billing_event_writing_enabled".
Who can share some knowledge on my following questions? if I enable this
property, when will the billing events be written? How long will the
billing events be kept?

Thanks

We have been trying to use the visual studio extension from https://github.com/cloudfoundry-incubator/cf-vs-extension but get an error when logging in.

Unexpected character encountered while parsing value: <. Path “. Line 0 position 0.

The error message in the uaa logs come from the filter called BackwardsCompatibleTokenEndpointAuthenticationFilter in the uaa code:

[2015-05-14 20:26:41.209] uaa - 14916 [http-bio-8080-exec-9] .... DEBUG --- BackwardsCompatibleTokenEndpointAuthenticationFilter: Authentication request for failed: org.springframework.security.authentication.BadCredentialsException: No client authentication found. Remember to put a filter upstream of the TokenEndpointAuthenticationFilter.
[2015-05-14 20:26:41.211] uaa - 14916 [http-bio-8080-exec-9] .... DEBUG --- DefaultOAuth2ExceptionRenderer: Written [error="unauthorized", error_description="No client authentication found. Remember to put a filter upstream of the TokenEndpointAuthenticationFilter."] as "application/json" using [org.springframework.http.converter.json.MappingJacksonHttpMessageConverter(a)60f269ce]

Has anyone else tried the cf-vs-extention on their private CF environments and have you seen this problem?


Jon Price
Intel Corp

On 15-05-14 10:23 AM, Daniel Jones wrote:

Thanks again for your input. Have you seen this problem with versions
of Tomcat before 8.0.20?

I don't have proper data gathered from older than 8.0.20, so I cannot
compare.
I was just wondering when did 8.0.20 become available in JBP, I found
this date:
HEAD https://download.run.pivotal.io/tomcat/tomcat-8.0.20.tar.gz | grep
Last-Modified
Last-Modified: Tue, 03 Mar 2015 11:35:19 GMT

David and I think we've narrowed down the issue to a change from using
Tomcat 8.0.18 to 8.0.21. We're running more tests and collaborating
with Pivotal support. We also noticed that non-prod versions of our
apps were taking longer to crash, so it would seem to be
activity-related at least.

Do you know how Tomcat's APR/NIO memory gets allocated? Is there a way
of telling from pmap whether pages are being used for NIO buffers or
by the APR?

I don't think you can get the info from pmap. The malloc_info xml shows
better allocation stats, but only stats.
Is Tomcat using APR library or NIO by default in CloudFoundry? I'd
assume that NIO isn't used by default.

Have you tried the "-Dsun.zip.disableMemoryMapping=true" JVM option to
rule out the possibility that zip/jar file access is causing the
trouble? There has been some bugs in the past in the JVM in that area:
http://javaeesupportpatterns.blogspot.com.es/2011/08/mmap-file-outofmemoryerror-and-pmap.html
. That has been fixed
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6280693 , but doing
a check with "-Dsun.zip.disableMemoryMapping=true" JVM option would be
interesting.
Mainly concerned about this commit:
https://github.com/apache/tomcat/commit/6e5420c67fbad81973d888ad3701a392fac4fc71

Since most commits weren't very interesting in this diff:
https://github.com/apache/tomcat/compare/075bc2d6...c0eb033f?w=1
Might make a difference to Jar file access. I'm not saying that this
commit is a problem, just seemed like a big change.

-Lari

There is no way to create custom roles.

Only the roles shown here are available:

http://docs.cloudfoundry.org/concepts/roles.html

CF Runtime Team

Hi Aaron,
ECP Support is a roadmap item at this time and doesn't have a set timeline.
Apart from adding ECP SAML SP support on the UAA side, the SAML IDP needs
to implement and support this profile as well.


Thanks,
Sree Tummidi
Sr. Product Manager
Identity - Pivotal Cloud Foundry


On Wed, May 13, 2015 at 5:03 PM, Huber, Aaron M <aaron.m.huber(a)intel.com>
wrote:

In our case we use email address as the username via LDAP as well (UPN
actually, but same thing) so it would be the same. Is there a timeline for
the ECP profile support?



Aaron



*From:* Filip Hanik [mailto:fhanik(a)pivotal.io]
*Sent:* Wednesday, May 13, 2015 3:54 PM
*To:* Mike Youngstrom
*Cc:* Sree Tummidi; Huber, Aaron M; CF Developers Mailing List

*Subject:* Re: [cf-dev] UAA, SAML, and LDAP questions



The problem with SAML is that we never see the username. We only receive
the username in form of an email address from the SAML IDP. This would not
correspond to the username you would log in to LDAP with.



The use case you describe would indicate we want two different
authentication sources represent the same authentication source.

I believe the correct solution here is to implement the SAML ECP profile.
At that point you'd have an option to go LDAP or SAML rather than trying to
mix both.



Filip





On Wed, May 13, 2015 at 3:30 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:

Possibly, though I think regular user authentication would still be a
concern for our users since security forces a rather short TTL for our
access tokens. I'll have to take a look and try a few things. We may
decide to just use LDAP and forget about the SSO integration for now.



Mike



On Wed, May 13, 2015 at 3:03 PM, Sree Tummidi <stummidi(a)pivotal.io> wrote:

Hi Aaron,

You could potentially use the access token (similar to a personal access
token used for GitHub API ) to achieve the CLI automation. The access token
can either be retrieved via an authentication to the CLI itself or via UAAC.

Regular users would still continue to use the -sso option.




Thanks,

Sree Tummidi

Sr. Product Manager

Identity - Pivotal Cloud Foundry





On Wed, May 13, 2015 at 1:56 PM, Huber, Aaron M <aaron.m.huber(a)intel.com>
wrote:

That’s the main concern we have as well – we currently need LDAP for the
CLI since SAML doesn’t work in that case, but we’d like SAML for web-based
interactions (SSO in a portal, etc.). But at present it seems like that’s
not possible without the user having to deal with effectively two separate
accounts.



Aaron



*From:* Mike Youngstrom [mailto:youngm(a)gmail.com]
*Sent:* Wednesday, May 13, 2015 1:34 PM
*To:* Filip Hanik
*Cc:* Huber, Aaron M; CF Developers Mailing List
*Subject:* Re: [cf-dev] UAA, SAML, and LDAP questions



Well, that's a bummer. Is there any way around that? Our SAML is backed
by the same LDAP so they are the same user. We can provide a unique ID to
correlate SAML with LDAP users.



Mike



On Wed, May 13, 2015 at 2:28 PM, Filip Hanik <fhanik(a)pivotal.io> wrote:

yes, it would result in two different shadow accounts, differentiated by
the value of the user's origin field







On Wed, May 13, 2015 at 2:08 PM, aaron_huber <aaron.m.huber(a)intel.com>
wrote:

Would the same user logging in via SAML and LDAP result in two different
UAA
user objects with different sources, so that the user would have two
different sets of orgs/spaces/apps?

Aaron



--
View this message in context:
http://cf-dev.70369.x6.nabble.com/cf-dev-UAA-SAML-and-LDAP-questions-tp62p65.html
Sent from the CF Dev mailing list archive at Nabble.com.

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev




_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

Hi,

We've had a look through this and think it would be useful to give our
perspective for now.

To clarify the Memory heuristics, they are just weightings and not
percentages, they were never meant to add up to a hundred. The fact the
default settings add up to 105 is purely chance. Will see if the docs can
be improved around this.

The fact the JRE isn't detecting an OOM error and triggering the
killjava.sh script and because increasing the native memory prolongs the
OOM, we believe the leak is occurring outside the JRE. NIO, JNI, lots of
threads creation, etc... .There are other things running in the container
that could be consuming the memory.

If you suspect Tomcat then 8.0.22 is out and has been published for use in
the Java Buildpack, moving up to that might help things. We still haven't
seen a rash of memory problems from the wider Tomcat community for either
8.0.20 or 8.0.22 though so this would be unexpected.

It is actually possible to set the MALLOC_ARENA_MAX using an environment
variable.

cf set-env <APP> MALLOC_ARENA_MAX 2

This can also be specified in an applications manifest file.

Finally, a change has gone in to the master branch of the Java buildpack
that moves all of the memory heuristics code to an external program written
in Go. This means that scaling a Java application with cf scale no longer
requires a separate restage for the new memory setting to be applied as the
Go code will calculate them during every application start. During staging
you will also be able to see the memory settings that have been calculated
come out on the console. The plan is to release this new feature in version
3.1 of the buildpack in a week or so time.

We do still have an issue on our backlog to look at this and it shouldn't
be too long before we get to it.
https://www.pivotaltracker.com/story/show/94381284

Chris.

*On Mon, May 11, 2015 at 13:19 PM, Lari Hotari <Lari at hotari.net
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>
**<mailto:Lari at hotari.net
<https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>>> wrote:*

fyi. Tomcat 8.0.20 might be consuming more memory than
8.0.18:https://github.com/cloudfoundry/java-buildpack/issues/166#issuecomment-94517568

* Other things we’ve tried:

*>>* - We set verbose garbage collection to verify there was no
*>* memory size issues within the JVM. There wasn’t.
*>>* - We tried setting minimum memory for native, it had no
*>* effect. The container still gets killed
*>>* - We tried adjusting the ‘memory heuristics’ so that they
*>* added up to 80 rather than 100. This had the effect of causing a delay
*>* in the container being killed. However it still was killed.
*>I think adjusting memory heuristics so that they add up to 80 doesn't
make a difference because the values aren't percentages.
The values are proportional weighting values used in the memory
calculation:https://github.com/grails-samples/java-buildpack/blob/b4abf89/docs/jre-oracle_jre.md#memory-calculation

I found out that the only way to reserve "unused" memory is to set a
high value for the native memory lower bound in the memory_sizes.native
setting of config/open_jdk_jre.yml .
Example:https://github.com/grails-samples/java-buildpack/blob/22e0f6a/config/open_jdk_jre.yml#L25

>>* This seems like classic memory leak behaviour to me.

*>In my case it wasn't a classical Java memory leak, since the Java
application wasn't leaking memory. I was able to confirm this by getting
some heap dumps with the HeapDumpServlet
(https://github.com/lhotari/java-buildpack-diagnostics-app/blob/master/src/main/groovy/io/github/lhotari/jbpdiagnostics/HeapDumpServlet.groovy)
and analyzing them.

In my case the JVM's RSS memory size is slowly growing. It probably is
some kind of memory leak since one process I've been monitoring now is
very close to the memory limit. The uptime is now almost 3 weeks.

Here is the latest diff of the meminfo report.https://gist.github.com/lhotari/ee77decc2585f56cf3ad#file-meminfo_diff_example2-txt

*From a Java perspective this isn't classical. The JVM heap isn't filling

*up. The problem is that RSS size is slowly growing and will eventually
cause the Java process to cross the memory boundary so that the process
gets kill by the Linux kernel cgroups OOM killer.

RSS size might be growing because of many reasons. I have been able to
slow down the growth by doing the various MALLOC_ and JVM parameter
tuning (-XX:MinMetaspaceExpansion=1M -XX:CodeCacheExpansionSize=1M). I'm
able to get a longer uptime, but the problem isn't solved.

Lari


On 15-05-11 06:41 AM, Head-Rapson, David wrote:

* Thanks for the continued advice.

*>> >>* We’ve hit on a key discovery after yet another a soak test this weekend.
*>>* - When we deploy using Tomcat 8.0.18 we don’t see the issue
*>>* - When we deploy using Tomcat 8.0.20 (same app version, same
*>* CF space, same services bound, same JBP code version, same JRE
*>* version, running at the same time), we see the crashes occurring after
*>* just a couple of hours.
*>> >>* Ideally we’d go ahead with the memory calculations you mentioned
*>* however we’re stuck on lucid64 because we’re using Pivotal CF 1.3.x &
*>* we’re having upgrade issues to 1.4.x.
*>>* So we’re not able to adjust MALLOC_ARENA_MAX, nor are we able to view
*>* RSS in pmap as you describe
*>> >>* Other things we’ve tried:
*>>* - We set verbose garbage collection to verify there was no
*>* memory size issues within the JVM. There wasn’t.
*>>* - We tried setting minimum memory for native, it had no
*>* effect. The container still gets killed
*>>* - We tried adjusting the ‘memory heuristics’ so that they
*>* added up to 80 rather than 100. This had the effect of causing a delay
*>* in the container being killed. However it still was killed.
*>> >>* This seems like classic memory leak behaviour to me.
*>> >>* *From:*Lari Hotari [mailto:lari.hotari at sagire.fi <https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>] *On Behalf Of *Lari
*>* Hotari
*>* *Sent:* 08 May 2015 16:25
*>* *To:* Daniel Jones; Head-Rapson, David
*>* *Cc:* cf-dev at lists.cloudfoundry.org <https://lists.cloudfoundry.org/mailman/listinfo/cf-dev>
*>* *Subject:* Re: [Cf-dev] [vcap-dev] Java OOM debugging
*>> >>>* For my case, it turned out to be essential to reserve enough memory
*>* for "native" in the JBP. For the 2GB total memory, I set the minimum
*>* to 330M. With that setting I have been able to get over 2 weeks up
*>* time by now.
*>>* I mentioned this in my previous email:
*>>* The workaround for that in my case was to add a native key under
*>* memory_sizes in open_jdk_jre.yml and set the minimum to 330M (that is
*>* for a 2GB total memory).
*>* see example
*>* https://github.com/grails-samples/java-buildpack/blob/22e0f6a/config/open_jdk_jre.yml#L25 <https://github.com/grails-samples/java-buildpack/blob/22e0f6a/config/open_jdk_jre.yml#L25>
*>* that was how I got the app I'm running on CF to stay within the memory
*>* bounds. I'm sure there is now also a way to get the keys without
*>* forking the buildpack. I could have also adjusted the percentage
*>* portions, but I wanted to set a hard minimum for this case.
*>>>* I've been trying to get some insight by diffing the reports gathered
*>* from the meminfo servlet
*>* https://github.com/lhotari/java-buildpack-diagnostics-app/blob/master/src/main/groovy/io/github/lhotari/jbpdiagnostics/MemoryInfoServlet.groovy <https://github.com/lhotari/java-buildpack-diagnostics-app/blob/master/src/main/groovy/io/github/lhotari/jbpdiagnostics/MemoryInfoServlet.groovy>
*>>>* Here is such an example of a diff:
*>* https://gist.github.com/lhotari/ee77decc2585f56cf3ad#file-meminfo_diff_example-txt <https://gist.github.com/lhotari/ee77decc2585f56cf3ad#file-meminfo_diff_example-txt>
*>>* meminfo has pmap output included to get the report of the memory map
*>* of the process. I have just noticed that most of the memory has
*>* already been mmap:ed from the OS and it's just growing in RSS size.
*>* For example:
*>* < 00000000a7600000 1471488 1469556 1469556 rw--- [ anon ]
*>* > 00000000a7600000 1471744 1470444 1470444 rw--- [ anon ]
*>>* The pmap output from lucid64 didn't include the RSS size, so you have
*>* to use cflinuxfs2 for this. It's also better because of other reasons.
*>* The glibc in lucid64 is old and has some bugs around the MALLOC_ARENA_MAX.
*>>* I was manually able to estimate the maximum size of the RSS size of
*>* what the Java process will consume by simply picking the large
*>* anon-blocks from the pmap report and calculating those blocks by the
*>* allocated virtual size (VSS).
*>* Based on this calculation, I picked the minimum of 330M for "native"
*>* in open_jdk_jre.yml as I mentioned before.
*>>* It looks like these rows are for the Heap size:
*>* < 00000000a7600000 1471488 1469556 1469556 rw--- [ anon ]
*>* > 00000000a7600000 1471744 1470444 1470444 rw--- [ anon ]
*>>* It looks like the JVM doesn't fully allocate that block in RSS
*>* initially and most of the growth of RSS size comes from that in my
*>* case. In your case, it might be something different.
*>>* I also added a servlet for getting glibc malloc_info statistics in XML
*>* format (). I haven't really analysed that information because of time
*>* constraints and because I don't have a pressing problem any more. btw.
*>* The malloc_info XML report is missing some key elements, that has been
*>* added in later glibc versions
*>* (https://github.com/bminor/glibc/commit/4d653a59ffeae0f46f76a40230e2cfa9587b7e7e <https://github.com/bminor/glibc/commit/4d653a59ffeae0f46f76a40230e2cfa9587b7e7e>).
*>>* If killjava.sh never fires and the app crashed with Warden out of
*>* memory errors, then I believe it's the kernel's cgroups OOM killer
*>* that has killed the container processes. I have found this location
*>* where Warden oom notifier gets the OOM notification event:
*>* https://github.com/cloudfoundry/warden/blob/ad18bff/warden/lib/warden/container/features/mem_limit.rb#L70 <https://github.com/cloudfoundry/warden/blob/ad18bff/warden/lib/warden/container/features/mem_limit.rb#L70>
*>* This is the oom.c source code:
*>* https://github.com/cloudfoundry/warden/blob/ad18bff7dc56acbc55ff10bcc6045ebdf0b20c97/warden/src/oom/oom.c <https://github.com/cloudfoundry/warden/blob/ad18bff7dc56acbc55ff10bcc6045ebdf0b20c97/warden/src/oom/oom.c>
*>* . It reads the cgroups control files and receives events from the
*>* kernel that way.
*>>* I'd suggest that you use pmap for the Java process after it has
*>* started and calculate the maximum RSS size by calculating the VSS size
*>* of the large anon blocks instead of RSS for the blocks that the Java
*>* process has reserved for it's different memory areas (I think you
*>* shouldn't . You should discard adding VSS for the
*>* CompressedClassSpaceSize block.
*>* After this calculation, add enough memory to the "native" parameter in
*>* JBP until the RSS size calculated this way stays under the limit.
*>* That's the only "method" I have come up by now.
*>>* It might be required to have some RSS space allocated for any zip/jar
*>* files read by the Java process. I think that Java uses mmap files for
*>* zip file reading by default and that might go on top of all other limits.
*>* To test this theory, I'd suggest testing by adding
*>* -Dsun.zip.disableMemoryMapping=true system property setting to
*>* JAVA_OPTS. That disables the native mmap for zip/jar file reading. I
*>* haven't had time to test this assumption.
*>>* I guess the only way to understand how Java allocates memory is to
*>* look at the source code.
*>* from http://openjdk.java.net/projects/jdk8u/ <http://openjdk.java.net/projects/jdk8u/> , the instructions to get
*>* the source code of JDK 8:
*>* hg clone http://hg.openjdk.java.net/jdk8u/jdk8u;cd <http://hg.openjdk.java.net/jdk8u/jdk8u;cd> jdk8u;sh get_source.sh
*>* This tool is really good for grepping and searching the source code:
*>* http://geoff.greer.fm/ag/ <http://geoff.greer.fm/ag/> <http://geoff.greer.fm/ag/ <http://geoff.greer.fm/ag/>>
*>* On Ubuntu it's in silversearcher-ag package, "apt-get install
*>* silversearcher-ag" and on MacOSX brew it's "brew install
*>* the_silver_searcher".
*>* This alias is pretty useful:
*>* alias codegrep='ag --color --group --pager less -C 5'
*>* Then you just search for the correct location in code by starting with
*>* the tokens you know about:
*>* codegrep MaxMetaspaceSize
*>* this gives pretty good starting points in looking how the JDK
*>* allocates memory.
*>>* So the JDK source code is only a few commands away.
*>>* It would be interesting to hear more about this if someone has the
*>* time to dig in to this. This is about how far I got and I hope sharing
*>* this information helps someone continue. :)
*>>>* Lari
*>* github/twitter: lhotari
*>

Truncated



--
--
Christopher Frost - GoPivotal UK