How to get files from crash apps


MaggieMeng
 

We encountered java out of memory issue in our CF env recently. We had enabled gc log for the application by setting JAVA_TOOL_OPTIONS: -Xloggc:gc.log. But when out of memory issue occurred, the application was stopped and removed. We can't get gc.log. Is there any way to keep the gc.log after application crashes? The CF version we are using is 197. Any advice would be appreciated.

Regards,
Maggie


James Bayer
 

if you're using DEA's i believe the container stays around on the DEA for
about an hour or so by default before being removed.

assuming you're on a DEA, to investigate, you'd need to know the DEA and
the warden handle to investigate, log in to the DEA as an operator with
bosh, then go the warden depot directory, find the warden handle and
execute bin/wsh from the container handle directory.

see:
http://docs.cloudfoundry.org/running/troubleshooting/troubleshooting-apps.html#access-warden

in diego, removing containers happens right away, but diego is also going
to support ssh for active troubleshooting and could just increase the
memory and attach jvm tools for debugging purposes to troubleshoot.

good luck.

On Tue, Jun 23, 2015 at 1:12 AM, Meng, Xiangyi <xiangyi.meng(a)emc.com> wrote:

We encountered java out of memory issue in our CF env recently. We had
enabled gc log for the application by setting JAVA_TOOL_OPTIONS:
-Xloggc:gc.log. But when out of memory issue occurred, the application was
stopped and removed. We can’t get gc.log. Is there any way to keep the
gc.log after application crashes? The CF version we are using is 197. Any
advice would be appreciated.



Regards,

Maggie

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Thank you,

James Bayer


MaggieMeng
 

I am using DEA. But I didn’t found where the container was kept. Is it compressed and stored somewhere? I also took a look at /var/vcap/data/dea_next/crashes/id, it only contain the files in “logs” directory: staging_task.log/stdout.log/stderr.log/env.log, not application files. We need the files under “app” directory. Also I tried to access warden as you said. But the warden under /var/vcap/data/warden/depot was removed immediately after application process was killed.



From: cf-dev-bounces(a)lists.cloudfoundry.org [mailto:cf-dev-bounces(a)lists.cloudfoundry.org] On Behalf Of James Bayer
Sent: 2015年6月24日 16:07
To: Discussions about Cloud Foundry projects and the system overall.
Subject: Re: [cf-dev] How to get files from crash apps

if you're using DEA's i believe the container stays around on the DEA for about an hour or so by default before being removed.

assuming you're on a DEA, to investigate, you'd need to know the DEA and the warden handle to investigate, log in to the DEA as an operator with bosh, then go the warden depot directory, find the warden handle and execute bin/wsh from the container handle directory.

see: http://docs.cloudfoundry.org/running/troubleshooting/troubleshooting-apps.html#access-warden

in diego, removing containers happens right away, but diego is also going to support ssh for active troubleshooting and could just increase the memory and attach jvm tools for debugging purposes to troubleshoot.

good luck.

On Tue, Jun 23, 2015 at 1:12 AM, Meng, Xiangyi <xiangyi.meng(a)emc.com<mailto:xiangyi.meng(a)emc.com>> wrote:
We encountered java out of memory issue in our CF env recently. We had enabled gc log for the application by setting JAVA_TOOL_OPTIONS: -Xloggc:gc.log. But when out of memory issue occurred, the application was stopped and removed. We can’t get gc.log. Is there any way to keep the gc.log after application crashes? The CF version we are using is 197. Any advice would be appreciated.

Regards,
Maggie

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev



--
Thank you,

James Bayer


James Bayer
 

i'm not sure if this is the right property or not, maybe dieu knows:
https://github.com/cloudfoundry/cf-release/blob/master/jobs/dea_next/spec#L65-L67

On Wed, Jun 24, 2015 at 7:35 PM, Meng, Xiangyi <xiangyi.meng(a)emc.com> wrote:

I am using DEA. But I didn’t found where the container was kept. Is it
compressed and stored somewhere? I also took a look at
/var/vcap/data/dea_next/crashes/id, it only contain the files in “logs”
directory: staging_task.log/stdout.log/stderr.log/env.log, not application
files. We need the files under “app” directory. Also I tried to access
warden as you said. But the warden under /var/vcap/data/warden/depot was
removed immediately after application process was killed.







*From:* cf-dev-bounces(a)lists.cloudfoundry.org [mailto:
cf-dev-bounces(a)lists.cloudfoundry.org] *On Behalf Of *James Bayer
*Sent:* 2015年6月24日 16:07
*To:* Discussions about Cloud Foundry projects and the system overall.
*Subject:* Re: [cf-dev] How to get files from crash apps



if you're using DEA's i believe the container stays around on the DEA for
about an hour or so by default before being removed.



assuming you're on a DEA, to investigate, you'd need to know the DEA and
the warden handle to investigate, log in to the DEA as an operator with
bosh, then go the warden depot directory, find the warden handle and
execute bin/wsh from the container handle directory.



see:
http://docs.cloudfoundry.org/running/troubleshooting/troubleshooting-apps.html#access-warden



in diego, removing containers happens right away, but diego is also going
to support ssh for active troubleshooting and could just increase the
memory and attach jvm tools for debugging purposes to troubleshoot.



good luck.



On Tue, Jun 23, 2015 at 1:12 AM, Meng, Xiangyi <xiangyi.meng(a)emc.com>
wrote:

We encountered java out of memory issue in our CF env recently. We had
enabled gc log for the application by setting JAVA_TOOL_OPTIONS:
-Xloggc:gc.log. But when out of memory issue occurred, the application was
stopped and removed. We can’t get gc.log. Is there any way to keep the
gc.log after application crashes? The CF version we are using is 197. Any
advice would be appreciated.



Regards,

Maggie


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev





--

Thank you,



James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Thank you,

James Bayer


Jack Cai
 

I shared one simple technique in this slide [1], see Page 20. Basically you
can add ";sleep 1d" to the start command of your application, to keep the
container around for one day. You can remove the "--no-route" option if
your application listens on the HTTP port in time during start up.

Jack

[1]
https://www.slideshare.net/greensight/10-common-errors-when-pushing-apps-to-cloud-foundry

On Thu, Jun 25, 2015 at 2:58 AM, James Bayer <jbayer(a)pivotal.io> wrote:

i'm not sure if this is the right property or not, maybe dieu knows:

https://github.com/cloudfoundry/cf-release/blob/master/jobs/dea_next/spec#L65-L67


On Wed, Jun 24, 2015 at 7:35 PM, Meng, Xiangyi <xiangyi.meng(a)emc.com>
wrote:

I am using DEA. But I didn’t found where the container was kept. Is it
compressed and stored somewhere? I also took a look at
/var/vcap/data/dea_next/crashes/id, it only contain the files in “logs”
directory: staging_task.log/stdout.log/stderr.log/env.log, not application
files. We need the files under “app” directory. Also I tried to access
warden as you said. But the warden under /var/vcap/data/warden/depot was
removed immediately after application process was killed.







*From:* cf-dev-bounces(a)lists.cloudfoundry.org [mailto:
cf-dev-bounces(a)lists.cloudfoundry.org] *On Behalf Of *James Bayer
*Sent:* 2015年6月24日 16:07
*To:* Discussions about Cloud Foundry projects and the system overall.
*Subject:* Re: [cf-dev] How to get files from crash apps



if you're using DEA's i believe the container stays around on the DEA for
about an hour or so by default before being removed.



assuming you're on a DEA, to investigate, you'd need to know the DEA and
the warden handle to investigate, log in to the DEA as an operator with
bosh, then go the warden depot directory, find the warden handle and
execute bin/wsh from the container handle directory.



see:
http://docs.cloudfoundry.org/running/troubleshooting/troubleshooting-apps.html#access-warden



in diego, removing containers happens right away, but diego is also going
to support ssh for active troubleshooting and could just increase the
memory and attach jvm tools for debugging purposes to troubleshoot.



good luck.



On Tue, Jun 23, 2015 at 1:12 AM, Meng, Xiangyi <xiangyi.meng(a)emc.com>
wrote:

We encountered java out of memory issue in our CF env recently. We had
enabled gc log for the application by setting JAVA_TOOL_OPTIONS:
-Xloggc:gc.log. But when out of memory issue occurred, the application was
stopped and removed. We can’t get gc.log. Is there any way to keep the
gc.log after application crashes? The CF version we are using is 197. Any
advice would be appreciated.



Regards,

Maggie


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev





--

Thank you,



James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Thank you,

James Bayer

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev