Re: Memory usage: number reported from `cf app <app>` vs. inside container


Julz Friedman
 

Hi John,

Unfortunately memory usage inside containers is a bit of a painful topic.
I've linked to a really great blog by Heroku on this topic below that goes
in to more detail but tl;dr there's not a great answer to your question
right now. Memory in the container is constrained by cgroups; things like
/proc/meminfo report based on the host rather than the cgroup and aren't
containerised.

If you're an operator and are on the host you can easily check the numbers
by finding your container's cgroup directory under /sys/fs/cgroup/memory/
and just reading the files in that directory (e.g. memory.stat,
memory.usage_in_bytes etc). Unfortunately we don't allow you to mount
cgroups in the container so you can't see this via cf ssh. When cgroups
namespaces are added to the kernel we may be able to change this but right
now the cgroup hierarchy is global so we don't want to expose it in
containers. Just to be clear though, what you see in the metrics cloud
foundry reports is identically the numbers from the host's cgroup
hierarchy, so the only way they can be wrong is if there's a kernel bug (in
which case you'd see the same numbers via cf ssh anyway). Garden never
kills your app itself for being out of memory, it just places it in a
cgroup with a limit and tells the kernel to kill it if it's exceeded.

Hope this was at least somewhat helpful!

[0]: https://fabiokung.com/2014/03/13/memory-inside-linux-containers/

On Mon, 31 Oct 2016 at 15:46, John Feminella <jxf(a)pivotal.io> wrote:

If I `cf ssh` into a (garden-runc) container, what can I run to convince
myself that the memory usage reported by `cf app <some-app>` is identical
to a number inside the container? Guardian says this is computed as follows
[0]:

stats.Memory.TotalUsageTowardLimit =
stats.Memory.TotalRss +
(stats.Memory.TotalCache - stats.Memory.TotalInactiveFile)

Can I accurately discover the values of TotalRss, TotalCache, and
TotalinactiveFile from inside the container? Could I look at /proc/meminfo,
for example, or is there a better way?

[0]
https://github.com/cloudfoundry/guardian/blob/e9346b9849a33cddb037628c79057c4c80d4e3d8/rundmc/runrunc/stats.go#L70

best,
~ jf
--
John Feminella
Advisory Platform Architect
✉ · jxf(a)pivotal.io
t · @jxxf

Join {cf-dev@lists.cloudfoundry.org to automatically receive all group messages.