Hi all,

Whilst investigating the Java Buildpack out-of-memory issues David
Head-Rapson mailed about the other day, we discovered a discrepancy between
the memory usage stat provided by `cf app` and the value stored in the
corresponding cgroup's `memory.usage_in_bytes` file. The latter seems to be
bumping right along the maximum allowed.


- We did a `cf app`, and got a memory stat of 847.6MiB of 896MiB.
- We got the appId from CF_TRACE, `bosh ssh`'d onto the right DEA
- We then did `cat
tmp/warden/cgroup/memory/instance-id/memory.usage_in_bytes` and got
939,515,904, which equates to 895.99ish MiB.

Does anyone know why the latter is so high, and why it would differ from
what the DEA reports back to the Cloud Controller? There's clearly a gap in
our understanding somewhere, so any help would be much appreciated.

Many thanks,

Daniel Jones
EngineerBetter.com

The reported statistic is calculated here:

https://github.com/cloudfoundry/dea_ng/blob/310797e1097dcd5531bff4077ccd8f02f6091219/lib/dea/stat_collector.rb#L92-L94

On Thu, May 7, 2015 at 8:28 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Hi all,

Whilst investigating the Java Buildpack out-of-memory issues David
Head-Rapson mailed about the other day, we discovered a discrepancy between
the memory usage stat provided by `cf app` and the value stored in the
corresponding cgroup's `memory.usage_in_bytes` file. The latter seems to be
bumping right along the maximum allowed.


- We did a `cf app`, and got a memory stat of 847.6MiB of 896MiB.
- We got the appId from CF_TRACE, `bosh ssh`'d onto the right DEA
- We then did `cat
tmp/warden/cgroup/memory/instance-id/memory.usage_in_bytes` and got
939,515,904, which equates to 895.99ish MiB.

Does anyone know why the latter is so high, and why it would differ from
what the DEA reports back to the Cloud Controller? There's clearly a gap in
our understanding somewhere, so any help would be much appreciated.

Many thanks,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Matthew Sykes
matthew.sykes(a)gmail.com

Thanks for the reply Matthew!

We saw that bit of code when we were following it through. When we go as
far as the call from the DEA's Warden client to the Warden Server, we
struggled to find where those stats (total_rss, total_cache,
total_inactive_file) came from. DO you happen to know what the source of
truth is for those data?

Thanks for your help.

On Thu, May 7, 2015 at 1:58 PM, Matthew Sykes <matthew.sykes(a)gmail.com>
wrote:

The reported statistic is calculated here:


https://github.com/cloudfoundry/dea_ng/blob/310797e1097dcd5531bff4077ccd8f02f6091219/lib/dea/stat_collector.rb#L92-L94

On Thu, May 7, 2015 at 8:28 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Hi all,

Whilst investigating the Java Buildpack out-of-memory issues David
Head-Rapson mailed about the other day, we discovered a discrepancy between
the memory usage stat provided by `cf app` and the value stored in the
corresponding cgroup's `memory.usage_in_bytes` file. The latter seems to be
bumping right along the maximum allowed.


- We did a `cf app`, and got a memory stat of 847.6MiB of 896MiB.
- We got the appId from CF_TRACE, `bosh ssh`'d onto the right DEA
- We then did `cat
tmp/warden/cgroup/memory/instance-id/memory.usage_in_bytes` and got
939,515,904, which equates to 895.99ish MiB.

Does anyone know why the latter is so high, and why it would differ from
what the DEA reports back to the Cloud Controller? There's clearly a gap in
our understanding somewhere, so any help would be much appreciated.

Many thanks,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Matthew Sykes
matthew.sykes(a)gmail.com

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Regards,

Daniel Jones
EngineerBetter.com

Aha - just found it, and it does indeed tally up. Thanks again!

On Thu, May 7, 2015 at 3:06 PM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Thanks for the reply Matthew!

We saw that bit of code when we were following it through. When we go as
far as the call from the DEA's Warden client to the Warden Server, we
struggled to find where those stats (total_rss, total_cache,
total_inactive_file) came from. DO you happen to know what the source of
truth is for those data?

Thanks for your help.

On Thu, May 7, 2015 at 1:58 PM, Matthew Sykes <matthew.sykes(a)gmail.com>
wrote:

The reported statistic is calculated here:


https://github.com/cloudfoundry/dea_ng/blob/310797e1097dcd5531bff4077ccd8f02f6091219/lib/dea/stat_collector.rb#L92-L94

On Thu, May 7, 2015 at 8:28 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Hi all,

Whilst investigating the Java Buildpack out-of-memory issues David
Head-Rapson mailed about the other day, we discovered a discrepancy between
the memory usage stat provided by `cf app` and the value stored in the
corresponding cgroup's `memory.usage_in_bytes` file. The latter seems to be
bumping right along the maximum allowed.


- We did a `cf app`, and got a memory stat of 847.6MiB of 896MiB.
- We got the appId from CF_TRACE, `bosh ssh`'d onto the right DEA
- We then did `cat
tmp/warden/cgroup/memory/instance-id/memory.usage_in_bytes` and got
939,515,904, which equates to 895.99ish MiB.

Does anyone know why the latter is so high, and why it would differ from
what the DEA reports back to the Cloud Controller? There's clearly a gap in
our understanding somewhere, so any help would be much appreciated.

Many thanks,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Matthew Sykes
matthew.sykes(a)gmail.com

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Regards,

Daniel Jones
EngineerBetter.com

--
Regards,

Daniel Jones
EngineerBetter.com