On 15-05-14 10:23 AM, Daniel Jones wrote:

Thanks again for your input. Have you seen this problem with versions
of Tomcat before 8.0.20?

I don't have proper data gathered from older than 8.0.20, so I cannot
compare.
I was just wondering when did 8.0.20 become available in JBP, I found
this date:
HEAD https://download.run.pivotal.io/tomcat/tomcat-8.0.20.tar.gz | grep
Last-Modified
Last-Modified: Tue, 03 Mar 2015 11:35:19 GMT

David and I think we've narrowed down the issue to a change from using
Tomcat 8.0.18 to 8.0.21. We're running more tests and collaborating
with Pivotal support. We also noticed that non-prod versions of our
apps were taking longer to crash, so it would seem to be
activity-related at least.

Do you know how Tomcat's APR/NIO memory gets allocated? Is there a way
of telling from pmap whether pages are being used for NIO buffers or
by the APR?

I don't think you can get the info from pmap. The malloc_info xml shows
better allocation stats, but only stats.
Is Tomcat using APR library or NIO by default in CloudFoundry? I'd
assume that NIO isn't used by default.

Have you tried the "-Dsun.zip.disableMemoryMapping=true" JVM option to
rule out the possibility that zip/jar file access is causing the
trouble? There has been some bugs in the past in the JVM in that area:
http://javaeesupportpatterns.blogspot.com.es/2011/08/mmap-file-outofmemoryerror-and-pmap.html
. That has been fixed
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6280693 , but doing
a check with "-Dsun.zip.disableMemoryMapping=true" JVM option would be
interesting.
Mainly concerned about this commit:
https://github.com/apache/tomcat/commit/6e5420c67fbad81973d888ad3701a392fac4fc71

Since most commits weren't very interesting in this diff:
https://github.com/apache/tomcat/compare/075bc2d6...c0eb033f?w=1
Might make a difference to Jar file access. I'm not saying that this
commit is a problem, just seemed like a big change.

-Lari