Hi,

CF runtime container and application memory usage
Do you know which one is accurate?

It is always hard to say what is actual memory usage because it depends on what we mean by it. And the meaning probably comes from the purpose.

Talking about CF it reports memory usage as RSS + CACHE - INACTIVE_FILE for a container. It is hard to say if it we can consider this being accurate or not but it is more or less what we can call "active memory", meaning that if we set memory limit less than that the application might become slower due to swapping out or using file i/o more often then required.

Some reference links:
- Linux Memory https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt
- DEA: https://github.com/cloudfoundry/dea_ng/blob/65b5415770a767dd38e18eeef7576b9c38aa3509/lib/dea/stat_collector.rb#L60
- Guardian: https://github.com/cloudfoundry/guardian/blob/cf6012a27dbf5646062a9a83f8c95d12a65c9856/rundmc/runrunc/stats.go#L70

Best regards,
Stanislav

________________________________
From: Zhiyong Li <Zhiyong.Li(a)sas.com>
Sent: Sunday, April 30, 2017 1:02
To: cf-dev(a)lists.cloudfoundry.org
Subject: [cf-dev] How to capture CF runtime container and application memory usage

We have used four different commands to capture the memory usage for the Springboot apps running in CF runtime including: jcmp, ps, cf app and pmap. However, everyone seems report different values. Do you know which one is accurate? Here are example outputs:

Running “cf apps” the memory usage is reported as 930.4M:

C:\Program Files\Cloud Foundry>cf app mtp-identities
Showing health and status for app mtp-identities in org stocf / space dev as admin...

name: mtp-identities
requested state: started
instances: 1/1
usage: 2G x 1 instances
routes: vdmml.17w21.sas.com/identities
last uploaded: Thu 27 Apr 19:01:05 EDT 2017
stack: cflinuxfs2
buildpack: java_buildpack

state since cpu memory disk details
#0 running 2017-04-27T23:02:13Z 0.2% 930.4M of 2G 186M of 2G


If I ssh into the container and run ‘ps’, the RSS size is reported as 939M:

vcap(a)dpgdg9915eg:~$ ps axo user,pid,cpu,rss,size,vsize,nlwp,cmd --sort -rss
USER PID CPU RSS SIZE VSZ NLWP CMD
vcap 13 - 962008 5909184 6023768 58 /home/vcap/app/.java-buildpack/open_jdk_jre/bin/java -Djava.io.tmpdir=/home/vcap/tmp -XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh -Xmx384M -XX:MaxMetaspaceSize=128M -Xss256K -Xms384M -XX:MetaspaceSize=128M -javaagent:/home/vcap/app/lib/aspectjweaver-1.8.9.jar -Dserver.port=8080 -Djava.security.egd=file:/dev/./urandom -Dconsul.token=tobeusedfordemosonlyclnt -XX:NativeMemoryTracking=detail -verbose:class -XX:MaxDirectMemorySize=10M -XX:ReservedCodeCacheSize=240M -cp /home/vcap/app/. org.springframework.boot.loader.JarLauncher

While still in the container, running pmap for the process, the RSS is reported as 944M, and the “dirty” column reports 921M:
Address Kbytes RSS Dirty Mode Mapping
…
total kB 6030924 966196 942984

None of this looks close to what is reported via the NMT (jcmp) totals though (644M of committed):

2017-04-28T12:28:03.81-0400 [APP/0] OUT Total: reserved=1889246KB +84548KB, committed=659678KB +137612KB