On Thu, May 14, 2015 at 2:59 PM, Lari Hotari <Lari(a)hotari.net> wrote:

On 15-05-14 10:23 AM, Daniel Jones wrote:

Thanks again for your input. Have you seen this problem with versions
of Tomcat before 8.0.20?

I don't have proper data gathered from older than 8.0.20, so I cannot
compare.
I was just wondering when did 8.0.20 become available in JBP, I found
this date:
HEAD https://download.run.pivotal.io/tomcat/tomcat-8.0.20.tar.gz | grep
Last-Modified
Last-Modified: Tue, 03 Mar 2015 11:35:19 GMT

David and I think we've narrowed down the issue to a change from using
Tomcat 8.0.18 to 8.0.21. We're running more tests and collaborating
with Pivotal support. We also noticed that non-prod versions of our
apps were taking longer to crash, so it would seem to be
activity-related at least.

Do you know how Tomcat's APR/NIO memory gets allocated? Is there a way
of telling from pmap whether pages are being used for NIO buffers or
by the APR?

I don't think you can get the info from pmap. The malloc_info xml shows
better allocation stats, but only stats.
Is Tomcat using APR library or NIO by default in CloudFoundry? I'd
assume that NIO isn't used by default.

Sorry for the duplicate. Resending as my first reply didn't go back to the
list.

Since the Connector in server.xml is not specifically setting an
implementation it should use the NIO connector, which is the default in
Tomcat 8. A quick test on PWS confirmed this for me.


https://github.com/cloudfoundry/java-buildpack/blob/master/resources/tomcat/conf/server.xml#L22

It would be interesting to see if the BIO or APR connectors have similar
issues. BIO would be easy to test, just add
`protocol="org.apache.coyote.http11.Http11Protocol"` to the Connector tag
on line #22.

APR would be trickier as you'd need to compile the native library and pull
that into the environment.

Dan

Have you tried the "-Dsun.zip.disableMemoryMapping=true" JVM option to
rule out the possibility that zip/jar file access is causing the
trouble? There has been some bugs in the past in the JVM in that area:

http://javaeesupportpatterns.blogspot.com.es/2011/08/mmap-file-outofmemoryerror-and-pmap.html
. That has been fixed
http://bugs.java.com/bugdatabase/view_bug.do?bug_id=6280693 , but doing
a check with "-Dsun.zip.disableMemoryMapping=true" JVM option would be
interesting.
Mainly concerned about this commit:

https://github.com/apache/tomcat/commit/6e5420c67fbad81973d888ad3701a392fac4fc71

Since most commits weren't very interesting in this diff:
https://github.com/apache/tomcat/compare/075bc2d6...c0eb033f?w=1
Might make a difference to Jar file access. I'm not saying that this
commit is a problem, just seemed like a big change.

-Lari
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev