Date   

Re: Post: Cloud Foundry Advisory Board Meeting Notes - 2015 June

Ferran Rodenas <frodenas@...>
 

Thanks Phil!

- Ferdy

2015-06-12 22:19 GMT+02:00 Phil Whelan <philw(a)activestate.com>:

Hi,

I've just posted a write-up of the Cloud Foundry Advisory Board Meeting
from Wednesday's meeting.

http://www.activestate.com/blog/2015/06/cloud-foundry-advisory-board-meeting-2015-june

Topics include...

- Foundation update
- Dojo Program
- UAA
- Runtime
- Core Services
- BOSH
- Services API
- Diego
- Loggregator
- Buildpacks
- Windows
- CLI
- Upcoming Events

Please, let me know if you spot any inaccuracies.

Cheers,
Phil

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: How random is Metron's Doppler selection?

John Tuley <jtuley@...>
 

I can't speculate as to the future of CF and whether or not a particular
feature will someday be included.

But I can suggest a workaround: aside from very paranoid application
security group settings, there should be nothing preventing your
application from sending syslog to your external drain already, since apps
can make outbound connections. Obviously, this wouldn't also go through
Loggregator, and so wouldn't be available on `cf logs`. But perhaps your
logging utility can be configured to send to both syslog and stdout/stderr
simultaneously?

– John Tuley

On Fri, Jun 12, 2015 at 2:40 PM, Mike Heath <elcapo(a)gmail.com> wrote:

That's fair.

I think Mike Youngstrom is right. All of our logging problems would go
away if our applications could talk syslog to Loggregator. Capturing stdout
and stderr is certainly convenient, but it's not great for dealing with
stack traces.

-Mike

On Fri, Jun 12, 2015 at 8:38 AM, John Tuley <jtuley(a)pivotal.io> wrote:

Mike,

I don't want to speak to the possibility, but I can explain why we
decided against app affinity. Basically, it comes down to sharding over a
dynamic pool. As Doppler instances come and go, Metron would need to
re-balance its affinity calculations. This becomes troublesome if you
assume that a single Doppler is responsible for each app (or app-instance),
including the recent history: does the old home of an app need to transfer
history to the new home? Or maybe a new server just picks up new apps, and
all the old mappings stay the same? We did some research into algorithms
for this sort of consistent hashing/sharding and determined that it would
be difficult to implement in the presence of distributed servers *and* distributed
clients.

Given that your goals don't include history, the problem becomes easier
for sure. But I'd (personally – not speaking for product leadership) be
wary of accepting a PR that only solved forward-rebalancing without
addressing the problem of historical data.

– John Tuley

On Thu, Jun 11, 2015 at 4:55 PM, Mike Heath <elcapo(a)gmail.com> wrote:

Actually, this might explain why some of our customers are so frustrated
trying to read their stack traces in Splunk. :\

So each line of a stack trace could go to a different Doppler. That
means each line of the stack trace goes out to a different syslog drain
making it impossible to consolidate that stack trace into a single logging
event when passed off to a third-party logging system like Splunk. This
sucks. To be fair, Splunk has never been any good at dealing with stack
traces.

What are the possibilities of getting some kind of optionally enabled
application instance affinity put into Metron? (I know. I know. I can
submit a PR.)

-Mike

On Thu, Jun 11, 2015 at 3:54 PM, John Tuley <jtuley(a)pivotal.io> wrote:

Oops, wrong link. Should be
https://github.com/cloudfoundry/loggregator/blob/develop/src/metron/main.go#L188-L197
.

Sorry about that!

– John Tuley

On Thu, Jun 11, 2015 at 3:36 PM, John Tuley <jtuley(a)pivotal.io> wrote:

Mike,

Metron chooses a randomly-available Doppler for each message
<https://www.pivotaltracker.com/story/show/96801752>. Availability
prefers same-zone Doppler servers:

- If a Metron instance knows about any same-zone Dopplers, it
chooses one at random for each message.
- If no same-zone Dopplers are present, the random choice is made
from the list of all known servers.


In fact, the behavior you describe is the behavior of DEA Logging
Agent before Metron existed. What we discovered with that approach is that
it balances load very unfairly, as a single high-volume app is stuck on one
server. While the "new" mechanism does not guarantee consistency, it does
enable the Doppler pool to more-evenly share load.

If you're seeing that a single app instance is routed to the same
Doppler server every time, then (without further information) I would guess
that you're either running a single Doppler instance in each availability
zone, or your deck is stacked. :-) If neither of those is true and you're
still observing that Metron routes messages from an app instance to a
single Doppler, I'd love to investigate how that is happening.

– John Tuley

On Thu, Jun 11, 2015 at 2:45 PM, Mike Heath <elcapo(a)gmail.com> wrote:

Metron's documentation [1] says "All Metron traffic is randomly
distributed across available Dopplers." How random is this? Based on
observation, it appears that logs for an individual application instance
are consistently sent to the same Doppler instance. The consistency aspect
is very important for us so that our Syslog forwarder can consolidate stack
traces into a single logging event.

How random is this distribution really for an application instance's
logs?

-Mike

1 -
https://github.com/cloudfoundry/loggregator/tree/develop/src/metron

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Log drain for an app

Daniel Mikusa
 

Thanks John! I'll play with that and see if I can make it work.

Dan

On Fri, Jun 12, 2015 at 4:26 PM, John Tuley <jtuley(a)pivotal.io> wrote:

Dan,

I see three questions in your email, which I'll try to address in turn:

- *"Can my application send logs with a unique token?"* – Your
application can add any text it likes to the message, of course. When
they're sent to the syslog drain, the messages will be embedded in a
syslog-formatted line. Looking at Logsene's syslog example
<https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Syslog#Syslog-Example>,
it seems that they expect the syslog message to contain a JSON payload with
the token as a property. If your application produces that JSON, I think it
would be compatible. However, the Loggregator system does not wrap bare
loglines into that format, nor can it be configured to do so without
rewriting code.
- *"Do multiple apps on CF send logs from the same IP address?"* –
Yes. But it's worse than that: not only do multiple app streams come from
the same IP address, but a single application's stream can come from
multiple IP addresses. So this is probably not good from Logsene's point of
view.
- *"Is Loggregator's HTTPS transport compatible with the ElasticSearch
API?"* – No. Loggregator makes a POST request to the HTTPS endpoint by
putting a syslog-formatted line into the body of the request. It does not
have support for building an ElasticSearch-compatible JSON payload around
the message.

It appears to me that the best shot you have of compatibility with Logsene
is having your application build messages in the expected way, with JSON
wrapper (if that's truly needed; my quick read of the syslog example I
linked above was unclear). Keep in mind that Loggregator sends each *line* separately,
so your JSON payload must be a single line to be transmitted correctly.

– John Tuley

On Fri, Jun 12, 2015 at 11:13 AM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

Hello,

I'm looking at sending logs from my app to Logsene [1] and I'm trying to
figure out if this is going to work. From their instructions it seems like
there are two possibilities: syslog & https

I'm not sure syslog will work as Logsene seems to either require a unique
token to be included in the syslog event or to have all syslog traffic from
my app come from one IP. I'm not sure that the first is possible, and the
second won't work as multiple apps on CF could send logs from the same IP
(, please correct me if I'm wrong on either point).

That leaves me with HTTPS. According to their docs, they support the
elasticsearch api [2] through which you can post events to them. It seems
to expect a JSON payload, with a standard format.

I see in the CF docs [3] that we support sending logs via HTTPS but it
doesn't really say how the information is sent via HTTPS. Does anyone know
if this will be compatible? and where I can find more information about
how we send log data via HTTPS?

Thanks

Dan


[1] -
https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Logsene+documentation%3A+Home
[2] -
https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Index+Events+via+Elasticsearch+API
[3] -
http://docs.cloudfoundry.org/devguide/services/log-management.html#config

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Post: Cloud Foundry Advisory Board Meeting Notes - 2015 June

Wayne E. Seguin
 

Thanks Phil!!!

On Fri, Jun 12, 2015 at 4:19 PM, Phil Whelan <philw(a)activestate.com> wrote:

Hi,

I've just posted a write-up of the Cloud Foundry Advisory Board Meeting
from Wednesday's meeting.

http://www.activestate.com/blog/2015/06/cloud-foundry-advisory-board-meeting-2015-june

Topics include...

- Foundation update
- Dojo Program
- UAA
- Runtime
- Core Services
- BOSH
- Services API
- Diego
- Loggregator
- Buildpacks
- Windows
- CLI
- Upcoming Events

Please, let me know if you spot any inaccuracies.

Cheers,
Phil

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Post: Cloud Foundry Advisory Board Meeting Notes - 2015 June

Dieu Cao <dcao@...>
 

Thanks Phil! As always, a great write up.

On Fri, Jun 12, 2015 at 1:19 PM, Phil Whelan <philw(a)activestate.com> wrote:

Hi,

I've just posted a write-up of the Cloud Foundry Advisory Board Meeting
from Wednesday's meeting.

http://www.activestate.com/blog/2015/06/cloud-foundry-advisory-board-meeting-2015-june

Topics include...

- Foundation update
- Dojo Program
- UAA
- Runtime
- Core Services
- BOSH
- Services API
- Diego
- Loggregator
- Buildpacks
- Windows
- CLI
- Upcoming Events

Please, let me know if you spot any inaccuracies.

Cheers,
Phil

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Syslog Drain to Logstash Problems

Josh Ghiloni
 

That did solve our problems. Thanks everyone!

Josh Ghiloni
Senior Consultant
303.932.2202 o | 303.590.5427 m | 303.565.2794 f
jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>

ECS Team
Technology Solutions Delivered
ECSTeam.com<http://ECSTeam.com>

On Jun 12, 2015, at 09:20, Josh Ghiloni <jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>> wrote:

It appears that we do not have the syslog_drain_binder templates as Mike mentioned — thanks guys! We’ll update that and give ‘er a go.

Josh Ghiloni
Senior Consultant
303.932.2202 o | 303.590.5427 m | 303.565.2794 f
jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>

ECS Team
Technology Solutions Delivered
ECSTeam.com<http://ecsteam.com/>





On Jun 11, 2015, at 11:10, John Tuley <jtuley(a)pivotal.io<mailto:jtuley(a)pivotal.io>> wrote:

Mike,

Thanks for finding that. I've filed a bug<https://www.pivotaltracker.com/story/show/96801752> to get the README fixed.

– John Tuley

On Thu, Jun 11, 2015 at 10:22 AM, Mike Jacobi <jacobi(a)adobe.com<mailto:jacobi(a)adobe.com>> wrote:
We had the same problem due to missing templates in our manifest.

We initially used the example manifest snippet shown at https://github.com/cloudfoundry/loggregator which mentions only the doppler template. Looking at https://github.com/cloudfoundry/loggregator/blob/develop/manifest-templates/cf-lamb.yml we later determined that we also needed the syslog_drain_binder and metron_agent templates for a complete loggregator deployment.

-Mike



On Jun 10, 2015, at 9:35 AM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote:

I was able submit a log entry from the loggregator VM using -

nc -w0 10.xx.xx.xx 5000 <<< "logging from loggregator"

and to test UDP

nc -u -w0 10.xx.xx.xx 5000 <<< "logging from loggregator"


Which leads me to believe the networking is working properly. Any other thoughts?
Thanks!
Steve

On Wed, Jun 3, 2015 at 6:14 PM, Josh Ghiloni <jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>> wrote:
We’ll check that, thanks!

Josh Ghiloni
Senior Consultant
303.932.2202<tel:303.932.2202> o | 303.590.5427<tel:303.590.5427> m | 303.565.2794<tel:303.565.2794> f
jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>

ECS Team
Technology Solutions Delivered
ECSTeam.com<http://ecsteam.com/>





On Jun 3, 2015, at 15:41, John Tuley <jtuley(a)pivotal.io<mailto:jtuley(a)pivotal.io>> wrote:

Steve,

Until recently (cf-release v198), binding a syslog service required restarting the app. If you're post-v198, it should Just Work.

However, one of the things that could be in your way is network security. In order to forward logs to your drain, your loggregator servers must be able to access that server. This is the most common cause we see of systems failing to forward to syslog drains.

Please let us know if you have more questions.

– John Tuley

On Wed, Jun 3, 2015 at 12:37 PM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote:
Hello,
We are having problems draining log messages to Logstash. The drain is setup as a user provided service.

cf cups logstash-drain -l syslog://xx.xx.xx.xx:5000

And then bound to the service.

cf bind-service myapp logstash-drain

But no log messages are coming through to Logstash. Or more specifically, we are using ELK and the messages aren't seen through Kibana.

We were able to log into the DEA and using netcat (nc), messages were successfully submitted to the ELK stack.

nc -w0 -u xx.xx.xx.xx 5000 <<< "logging from remote"

Any suggestions on how to debug this further?
-Steve


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: How random is Metron's Doppler selection?

Mike Heath
 

That's fair.

I think Mike Youngstrom is right. All of our logging problems would go away
if our applications could talk syslog to Loggregator. Capturing stdout and
stderr is certainly convenient, but it's not great for dealing with stack
traces.

-Mike

On Fri, Jun 12, 2015 at 8:38 AM, John Tuley <jtuley(a)pivotal.io> wrote:

Mike,

I don't want to speak to the possibility, but I can explain why we decided
against app affinity. Basically, it comes down to sharding over a dynamic
pool. As Doppler instances come and go, Metron would need to re-balance its
affinity calculations. This becomes troublesome if you assume that a single
Doppler is responsible for each app (or app-instance), including the recent
history: does the old home of an app need to transfer history to the new
home? Or maybe a new server just picks up new apps, and all the old
mappings stay the same? We did some research into algorithms for this sort
of consistent hashing/sharding and determined that it would be difficult to
implement in the presence of distributed servers *and* distributed
clients.

Given that your goals don't include history, the problem becomes easier
for sure. But I'd (personally – not speaking for product leadership) be
wary of accepting a PR that only solved forward-rebalancing without
addressing the problem of historical data.

– John Tuley

On Thu, Jun 11, 2015 at 4:55 PM, Mike Heath <elcapo(a)gmail.com> wrote:

Actually, this might explain why some of our customers are so frustrated
trying to read their stack traces in Splunk. :\

So each line of a stack trace could go to a different Doppler. That means
each line of the stack trace goes out to a different syslog drain making it
impossible to consolidate that stack trace into a single logging event when
passed off to a third-party logging system like Splunk. This sucks. To be
fair, Splunk has never been any good at dealing with stack traces.

What are the possibilities of getting some kind of optionally enabled
application instance affinity put into Metron? (I know. I know. I can
submit a PR.)

-Mike

On Thu, Jun 11, 2015 at 3:54 PM, John Tuley <jtuley(a)pivotal.io> wrote:

Oops, wrong link. Should be
https://github.com/cloudfoundry/loggregator/blob/develop/src/metron/main.go#L188-L197
.

Sorry about that!

– John Tuley

On Thu, Jun 11, 2015 at 3:36 PM, John Tuley <jtuley(a)pivotal.io> wrote:

Mike,

Metron chooses a randomly-available Doppler for each message
<https://www.pivotaltracker.com/story/show/96801752>. Availability
prefers same-zone Doppler servers:

- If a Metron instance knows about any same-zone Dopplers, it
chooses one at random for each message.
- If no same-zone Dopplers are present, the random choice is made
from the list of all known servers.


In fact, the behavior you describe is the behavior of DEA Logging Agent
before Metron existed. What we discovered with that approach is that it
balances load very unfairly, as a single high-volume app is stuck on one
server. While the "new" mechanism does not guarantee consistency, it does
enable the Doppler pool to more-evenly share load.

If you're seeing that a single app instance is routed to the same
Doppler server every time, then (without further information) I would guess
that you're either running a single Doppler instance in each availability
zone, or your deck is stacked. :-) If neither of those is true and you're
still observing that Metron routes messages from an app instance to a
single Doppler, I'd love to investigate how that is happening.

– John Tuley

On Thu, Jun 11, 2015 at 2:45 PM, Mike Heath <elcapo(a)gmail.com> wrote:

Metron's documentation [1] says "All Metron traffic is randomly
distributed across available Dopplers." How random is this? Based on
observation, it appears that logs for an individual application instance
are consistently sent to the same Doppler instance. The consistency aspect
is very important for us so that our Syslog forwarder can consolidate stack
traces into a single logging event.

How random is this distribution really for an application instance's
logs?

-Mike

1 -
https://github.com/cloudfoundry/loggregator/tree/develop/src/metron

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Log drain for an app

John Tuley <jtuley@...>
 

Dan,

I see three questions in your email, which I'll try to address in turn:

- *"Can my application send logs with a unique token?"* – Your
application can add any text it likes to the message, of course. When
they're sent to the syslog drain, the messages will be embedded in a
syslog-formatted line. Looking at Logsene's syslog example
<https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Syslog#Syslog-Example>,
it seems that they expect the syslog message to contain a JSON payload with
the token as a property. If your application produces that JSON, I think it
would be compatible. However, the Loggregator system does not wrap bare
loglines into that format, nor can it be configured to do so without
rewriting code.
- *"Do multiple apps on CF send logs from the same IP address?"* – Yes.
But it's worse than that: not only do multiple app streams come from the
same IP address, but a single application's stream can come from multiple
IP addresses. So this is probably not good from Logsene's point of view.
- *"Is Loggregator's HTTPS transport compatible with the ElasticSearch
API?"* – No. Loggregator makes a POST request to the HTTPS endpoint by
putting a syslog-formatted line into the body of the request. It does not
have support for building an ElasticSearch-compatible JSON payload around
the message.

It appears to me that the best shot you have of compatibility with Logsene
is having your application build messages in the expected way, with JSON
wrapper (if that's truly needed; my quick read of the syslog example I
linked above was unclear). Keep in mind that Loggregator sends each
*line* separately,
so your JSON payload must be a single line to be transmitted correctly.

– John Tuley

On Fri, Jun 12, 2015 at 11:13 AM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

Hello,

I'm looking at sending logs from my app to Logsene [1] and I'm trying to
figure out if this is going to work. From their instructions it seems like
there are two possibilities: syslog & https

I'm not sure syslog will work as Logsene seems to either require a unique
token to be included in the syslog event or to have all syslog traffic from
my app come from one IP. I'm not sure that the first is possible, and the
second won't work as multiple apps on CF could send logs from the same IP
(, please correct me if I'm wrong on either point).

That leaves me with HTTPS. According to their docs, they support the
elasticsearch api [2] through which you can post events to them. It seems
to expect a JSON payload, with a standard format.

I see in the CF docs [3] that we support sending logs via HTTPS but it
doesn't really say how the information is sent via HTTPS. Does anyone know
if this will be compatible? and where I can find more information about
how we send log data via HTTPS?

Thanks

Dan


[1] -
https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Logsene+documentation%3A+Home
[2] -
https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Index+Events+via+Elasticsearch+API
[3] -
http://docs.cloudfoundry.org/devguide/services/log-management.html#config

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Post: Cloud Foundry Advisory Board Meeting Notes - 2015 June

Phil Whelan
 

Hi,

I've just posted a write-up of the Cloud Foundry Advisory Board Meeting
from Wednesday's meeting.
http://www.activestate.com/blog/2015/06/cloud-foundry-advisory-board-meeting-2015-june

Topics include...

- Foundation update
- Dojo Program
- UAA
- Runtime
- Core Services
- BOSH
- Services API
- Diego
- Loggregator
- Buildpacks
- Windows
- CLI
- Upcoming Events

Please, let me know if you spot any inaccuracies.

Cheers,
Phil


Log drain for an app

Daniel Mikusa
 

Hello,

I'm looking at sending logs from my app to Logsene [1] and I'm trying to
figure out if this is going to work. From their instructions it seems like
there are two possibilities: syslog & https

I'm not sure syslog will work as Logsene seems to either require a unique
token to be included in the syslog event or to have all syslog traffic from
my app come from one IP. I'm not sure that the first is possible, and the
second won't work as multiple apps on CF could send logs from the same IP
(, please correct me if I'm wrong on either point).

That leaves me with HTTPS. According to their docs, they support the
elasticsearch api [2] through which you can post events to them. It seems
to expect a JSON payload, with a standard format.

I see in the CF docs [3] that we support sending logs via HTTPS but it
doesn't really say how the information is sent via HTTPS. Does anyone know
if this will be compatible? and where I can find more information about
how we send log data via HTTPS?

Thanks

Dan


[1] -
https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Logsene+documentation%3A+Home
[2] -
https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Index+Events+via+Elasticsearch+API
[3] -
http://docs.cloudfoundry.org/devguide/services/log-management.html#config


Re: Syslog Drain to Logstash Problems

Josh Ghiloni
 

It appears that we do not have the syslog_drain_binder templates as Mike mentioned — thanks guys! We’ll update that and give ‘er a go.

Josh Ghiloni
Senior Consultant
303.932.2202 o | 303.590.5427 m | 303.565.2794 f
jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>

ECS Team
Technology Solutions Delivered
ECSTeam.com<http://ECSTeam.com>

On Jun 11, 2015, at 11:10, John Tuley <jtuley(a)pivotal.io<mailto:jtuley(a)pivotal.io>> wrote:

Mike,

Thanks for finding that. I've filed a bug<https://www.pivotaltracker.com/story/show/96801752> to get the README fixed.

– John Tuley

On Thu, Jun 11, 2015 at 10:22 AM, Mike Jacobi <jacobi(a)adobe.com<mailto:jacobi(a)adobe.com>> wrote:
We had the same problem due to missing templates in our manifest.

We initially used the example manifest snippet shown at https://github.com/cloudfoundry/loggregator which mentions only the doppler template. Looking at https://github.com/cloudfoundry/loggregator/blob/develop/manifest-templates/cf-lamb.yml we later determined that we also needed the syslog_drain_binder and metron_agent templates for a complete loggregator deployment.

-Mike



On Jun 10, 2015, at 9:35 AM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote:

I was able submit a log entry from the loggregator VM using -

nc -w0 10.xx.xx.xx 5000 <<< "logging from loggregator"

and to test UDP

nc -u -w0 10.xx.xx.xx 5000 <<< "logging from loggregator"


Which leads me to believe the networking is working properly. Any other thoughts?
Thanks!
Steve

On Wed, Jun 3, 2015 at 6:14 PM, Josh Ghiloni <jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>> wrote:
We’ll check that, thanks!

Josh Ghiloni
Senior Consultant
303.932.2202<tel:303.932.2202> o | 303.590.5427<tel:303.590.5427> m | 303.565.2794<tel:303.565.2794> f
jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>

ECS Team
Technology Solutions Delivered
ECSTeam.com<http://ecsteam.com/>





On Jun 3, 2015, at 15:41, John Tuley <jtuley(a)pivotal.io<mailto:jtuley(a)pivotal.io>> wrote:

Steve,

Until recently (cf-release v198), binding a syslog service required restarting the app. If you're post-v198, it should Just Work.

However, one of the things that could be in your way is network security. In order to forward logs to your drain, your loggregator servers must be able to access that server. This is the most common cause we see of systems failing to forward to syslog drains.

Please let us know if you have more questions.

– John Tuley

On Wed, Jun 3, 2015 at 12:37 PM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote:
Hello,
We are having problems draining log messages to Logstash. The drain is setup as a user provided service.

cf cups logstash-drain -l syslog://xx.xx.xx.xx:5000

And then bound to the service.

cf bind-service myapp logstash-drain

But no log messages are coming through to Logstash. Or more specifically, we are using ELK and the messages aren't seen through Kibana.

We were able to log into the DEA and using netcat (nc), messages were successfully submitted to the ELK stack.

nc -w0 -u xx.xx.xx.xx 5000 <<< "logging from remote"

Any suggestions on how to debug this further?
-Steve


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: How random is Metron's Doppler selection?

John Tuley <jtuley@...>
 

Mike,

I don't want to speak to the possibility, but I can explain why we decided
against app affinity. Basically, it comes down to sharding over a dynamic
pool. As Doppler instances come and go, Metron would need to re-balance its
affinity calculations. This becomes troublesome if you assume that a single
Doppler is responsible for each app (or app-instance), including the recent
history: does the old home of an app need to transfer history to the new
home? Or maybe a new server just picks up new apps, and all the old
mappings stay the same? We did some research into algorithms for this sort
of consistent hashing/sharding and determined that it would be difficult to
implement in the presence of distributed servers *and* distributed clients.

Given that your goals don't include history, the problem becomes easier for
sure. But I'd (personally – not speaking for product leadership) be wary of
accepting a PR that only solved forward-rebalancing without addressing the
problem of historical data.

– John Tuley

On Thu, Jun 11, 2015 at 4:55 PM, Mike Heath <elcapo(a)gmail.com> wrote:

Actually, this might explain why some of our customers are so frustrated
trying to read their stack traces in Splunk. :\

So each line of a stack trace could go to a different Doppler. That means
each line of the stack trace goes out to a different syslog drain making it
impossible to consolidate that stack trace into a single logging event when
passed off to a third-party logging system like Splunk. This sucks. To be
fair, Splunk has never been any good at dealing with stack traces.

What are the possibilities of getting some kind of optionally enabled
application instance affinity put into Metron? (I know. I know. I can
submit a PR.)

-Mike

On Thu, Jun 11, 2015 at 3:54 PM, John Tuley <jtuley(a)pivotal.io> wrote:

Oops, wrong link. Should be
https://github.com/cloudfoundry/loggregator/blob/develop/src/metron/main.go#L188-L197
.

Sorry about that!

– John Tuley

On Thu, Jun 11, 2015 at 3:36 PM, John Tuley <jtuley(a)pivotal.io> wrote:

Mike,

Metron chooses a randomly-available Doppler for each message
<https://www.pivotaltracker.com/story/show/96801752>. Availability
prefers same-zone Doppler servers:

- If a Metron instance knows about any same-zone Dopplers, it
chooses one at random for each message.
- If no same-zone Dopplers are present, the random choice is made
from the list of all known servers.


In fact, the behavior you describe is the behavior of DEA Logging Agent
before Metron existed. What we discovered with that approach is that it
balances load very unfairly, as a single high-volume app is stuck on one
server. While the "new" mechanism does not guarantee consistency, it does
enable the Doppler pool to more-evenly share load.

If you're seeing that a single app instance is routed to the same
Doppler server every time, then (without further information) I would guess
that you're either running a single Doppler instance in each availability
zone, or your deck is stacked. :-) If neither of those is true and you're
still observing that Metron routes messages from an app instance to a
single Doppler, I'd love to investigate how that is happening.

– John Tuley

On Thu, Jun 11, 2015 at 2:45 PM, Mike Heath <elcapo(a)gmail.com> wrote:

Metron's documentation [1] says "All Metron traffic is randomly
distributed across available Dopplers." How random is this? Based on
observation, it appears that logs for an individual application instance
are consistently sent to the same Doppler instance. The consistency aspect
is very important for us so that our Syslog forwarder can consolidate stack
traces into a single logging event.

How random is this distribution really for an application instance's
logs?

-Mike

1 - https://github.com/cloudfoundry/loggregator/tree/develop/src/metron

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Java Buildpack: Stale Files Outside of WEB-INF

Daniel Mikusa
 

On Fri, Jun 12, 2015 at 10:02 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Hi Dan,

Thanks for getting back to me.

Sadly I can't reproduce the issue on my local Cloud Foundry instance, so I
suspect the issue has something to do with the state of the client's CF's
blob store, their version of CF, or some peculiarity of the .war.

The behaviour I'm seeing is quite odd.

There is a subdirectory called "conf" in the .war file.
Ok. I don't think that should cause a problem.


If I make any changes inside this directory, they are not reflected after
an app push,
That's weird, but might happen. Some thoughts on how to see what's
happening. Apologizes in advance as some of this might be pretty basic.

- check that the WAR file generated includes those changes
- check that you're pointing to the right WAR file, i.e. that `-p` or
path in the manifest is pointing to the right location
- check that cf is not skipping the files for some reason (resource
matching). It shouldn't be an issue with files under 65k, which are always
pushed. In the CF_TRACE output, you should see the hashes of the files
that are sent to the server. I'd suggest hashing the file manually and
making sure it matches. Just to confirm everything is OK there.
- check the build pack that you're using. If it's been customized, make
sure it's not doing anything with that directory, like perhaps trying to
dynamically modify the config or something that could overwrite your local
changes.
- Modify the build pack so that the first thing it does is to hash the
files in conf and prints the hashes to the screen. That would let you see
the hash of the file that's given to the build pack and help to narrow down
where that change might happen.


even if I delete the app beforehand.
If you delete the app, that should delete any state associated with it.
The next push should include the most recent set of files, as CF wouldn't
have any other files for the app. Since it seems like this does not help,
it makes me thing that something else is modifying or overwriting your
files. Perhaps in the build pack or when the app starts.


If I make changes anywhere else in the .war, the changes are take effect.
If I rename "conf" to "conf-new", the change *is* effected.
I don't think that the official JBP would do anything with a "conf"
directory. It stores it's internal stuff in a hidden folder called
`.java-buildpack/`. It is suspicious that it's a "conf" directory though.
Perhaps something in a custom build pack or in the app is overwriting
this? Perhaps to auto configure it to use a bound service or something
like that?



I've tried this on two completely separate Cloud Foundry instances (PCF
1.3.3) and get the same result. Both have 'seen' older versions of the war.
That could be possible, if it's something in the app or build pack that's
overwriting that conf directory or it's contents.



CF_TRACE shows that in the resource matching request all 12,000-odd files
are submitted. The response is significantly smaller, and does not include
any of the files from the "conf" directory.
That sounds OK. I believe that returns anything that matches and is larger
than the defined threshold, which defaults to 65k.



Any suggestions on where to look for more clues?
See above. If that doesn't work, open a support ticket w/Pivotal Support
since this is happening on PCF and we can do some interactive debugging.

Dan





On Thu, Jun 11, 2015 at 2:20 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

On Thu, Jun 11, 2015 at 7:42 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Hi all,

If one pushes a .war containing files outside of WEB-INF, deletes the
app, then pushes a new version where those files outside of WEB-INF have
been changed, these changes are not reflected in the running app. Is this
expected behaviour? The buildpack is forked, from commit 71ab82a
<https://github.com/cloudfoundry/java-buildpack/commit/71ab82af4cd95465dcbc35eb89f3ac9dc9431e9a>
.
Seems odd. Every time you stage an app, you should get a copy of the
latest app bits (i.e. what was pushed up by cf). If you're not getting the
latest set of files, I don't think that would be a build pack problem. The
DEA should provide the build pack with the latest set of your files, if
it's not I don't think there's much the build pack can do about it.


I was just about to go about creating a repeatable test case and raising
a GitHub issue, but figured it'd be best to check that this isn't the way
it's *supposed* to work.
A test case would be a good idea. You could post it here, I'd be happy
to play around with it. Steps to repro would also be helpful.

Dan



For the record I realise that resources should be in WEB-INF by
convention, but the users where I am are expecting feature parity with
non-CF Tomcat which unpacks the whole .war.

Many thanks in advance,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


--
Regards,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: Java Buildpack: Stale Files Outside of WEB-INF

Daniel Jones
 

Hi Dan,

Thanks for getting back to me.

Sadly I can't reproduce the issue on my local Cloud Foundry instance, so I
suspect the issue has something to do with the state of the client's CF's
blob store, their version of CF, or some peculiarity of the .war.

The behaviour I'm seeing is quite odd.

There is a subdirectory called "conf" in the .war file. If I make any
changes inside this directory, they are not reflected after an app push,
even if I delete the app beforehand. If I make changes anywhere else in the
.war, the changes are take effect. If I rename "conf" to "conf-new", the
change *is* effected.

I've tried this on two completely separate Cloud Foundry instances (PCF
1.3.3) and get the same result. Both have 'seen' older versions of the war.

CF_TRACE shows that in the resource matching request all 12,000-odd files
are submitted. The response is significantly smaller, and does not include
any of the files from the "conf" directory.

Any suggestions on where to look for more clues?

On Thu, Jun 11, 2015 at 2:20 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

On Thu, Jun 11, 2015 at 7:42 AM, Daniel Jones <
daniel.jones(a)engineerbetter.com> wrote:

Hi all,

If one pushes a .war containing files outside of WEB-INF, deletes the
app, then pushes a new version where those files outside of WEB-INF have
been changed, these changes are not reflected in the running app. Is this
expected behaviour? The buildpack is forked, from commit 71ab82a
<https://github.com/cloudfoundry/java-buildpack/commit/71ab82af4cd95465dcbc35eb89f3ac9dc9431e9a>
.
Seems odd. Every time you stage an app, you should get a copy of the
latest app bits (i.e. what was pushed up by cf). If you're not getting the
latest set of files, I don't think that would be a build pack problem. The
DEA should provide the build pack with the latest set of your files, if
it's not I don't think there's much the build pack can do about it.


I was just about to go about creating a repeatable test case and raising
a GitHub issue, but figured it'd be best to check that this isn't the way
it's *supposed* to work.
A test case would be a good idea. You could post it here, I'd be happy to
play around with it. Steps to repro would also be helpful.

Dan



For the record I realise that resources should be in WEB-INF by
convention, but the users where I am are expecting feature parity with
non-CF Tomcat which unpacks the whole .war.

Many thanks in advance,

Daniel Jones
EngineerBetter.com

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

--
Regards,

Daniel Jones
EngineerBetter.com


(No subject)

Daniel Mikusa
 

For what it's worth, these aren't logs from Jetty. They're from the
gorouter, note the "[RTR]" tag. Logs from your app would be prefixed with
"[APP/#]". You might want to look around with `cf files <app> <path>` and
see if the app is writing log files to the local disk instead.

Dan

On Fri, Jun 12, 2015 at 4:21 AM, Arbi Akhina <arbi.akhina(a)gmail.com> wrote:

Yeah, I get Jetty logs as below:
2015-06-11T04:27:46.34+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:27:46
+0000] "GET / HTTP/1.1" 200 1098 "-" "curl/7.29.0" 10.0.2.15:58442
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:a94d78ae-81a8-4ca4-55f9-9ce8fe9d8f7b
response_time:0.027328613 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f
2015-06-11T04:42:23.03+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:42:23
+0000] "GET / HTTP/1.1" 200 1098 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Dragon/42.2.2.138
Chrome/42.0.2311.135 Safari/537.36" 10.0.2.15:32833
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:e24de0c2-d81c-4cd6-76cc-7077bb5ec26c
response_time:0.024409247 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f
2015-06-11T04:42:23.29+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:42:23
+0000] "GET /favicon.ico HTTP/1.1" 200 1098 "
http://modules-popeyed-nonascetic.10.244.0.34.xip.io/" "Mozilla/5.0
(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Dragon/
42.2.2.138 Chrome/42.0.2311.135 Safari/537.36" 10.0.2.15:32833
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:b17c21b4-eba7-4fd5-4fac-4826f89feaae
response_time:0.023473933 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f


On Thu, Jun 11, 2015 at 5:48 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

On Thu, Jun 11, 2015 at 11:37 AM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

I removed static assets as well as many classes that were not used,
I left classes related to jetty, javax (managment, ..), apache commons
io, and the package (with all classes) where the main class is located.
Interesting. Do you get log output from the application when it's
working? I'm curious because you weren't seeing this when it was failing.
Trying to understand if the app is not logging properly or if your logs
were being lost somehow.



I didn't try on public cfs like PWS (what is this one?) and BlueMix.
Pivotal Web Services. Similar to Bluemix, it's a public CF that you can
sign up to use. There are others too. The idea here is just to test this
on a larger, more powerful system and see if the problems go away.

Dan




On Thu, Jun 11, 2015 at 4:42 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Thu, Jun 11, 2015 at 10:31 AM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

Hi, I had to reduce the size of the jar to ~ 51M to get the simple
class that runs jetty to work on bosh-lite!!!
Interesting. Out of curiosity what did you remove to make it smaller?
Static assets? Also have you tried a larger CF install like PWS / BlueMix,
just to see if this is an issue specific to bosh-lite.


Is there any kind of size restriction for executable jars??
The system can put a limit on how big of an application you can
upload. I don't know about bosh-lite specifically, but the default is 1G.
Also, when you exceed that you'll get a message from the cf cli saying your
app is too big. It won't silently fail, which seems to be what happened
here.

Dan



Here is the system information of the warden container on which the
jar is running:
Java version : 1.8.0_45-
Java Spec version : 1.8
JVM version : 25.45-b02
JVM vendor : Oracle Corporation
JVM name : OpenJDK 64-Bit Server VM
Os Name : Linux
Os Architecture : amd64
Os version : 3.13.0-44-generic
Number of processors : 4
Max memory : 3087007744
Total memory : 3087007744
Available memory : 2958157616
Free work disk space : 17898352640
VM Args : [-Djava.io.tmpdir=/home/vcap/tmp,
-XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh,
-Xss1M, -Xmx3G, -Xms3G, -XX:MaxMetaspaceSize=419430K,
-XX:MetaspaceSize=419430K]

On Tue, Jun 9, 2015 at 7:01 PM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

no I meant the source is for the company I work for, not my personal
project.

On Tue, Jun 9, 2015 at 6:58 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Tue, Jun 9, 2015 at 12:43 PM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

I don't own the source, but here is the jar file
http://tempsend.com/6D745B7B07/7033/modules.jar
you can try to run it locally with java -jar modules.jar then
connect to localhost:8080
Hmm, OK. I thought you had a minimal example from you last note.
I'm not really up for running untrusted binary code. If you can put a
minimal code sample together, I'll give it a try.



An here is the manifest.yml i'm using:
---
applications:
- name: modules
memory: 4G
disk_quota: 2G
timeout: 180
instances: 1
host: modules-${random-word}
path: modules.jar
buildpack: https://github.com/cloudfoundry/java-buildpack
command: sleep 2 &&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY com.heavenize.osmoze.kernel.HelloHandler
# env:
# JAVA_OPTS: "$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000"

I'm having both files in the same folder modules/ and I push with cf
push
That seems OK. Is this app designed to run on CF? By that, I mean
is it configure to log to STDOUT / STDERR and is it going to listen on the
right port (i.e $PORT)?

Dan




Thanks a lot

On Tue, Jun 9, 2015 at 6:25 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

I'd be happy to try the demo and see if it works for me. Can you
post a complete project on github so I can just `git clone` & build?

Dan

On Tue, Jun 9, 2015 at 12:21 PM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

I've tried with a simple app that just launches an embedded jetty
that listen on the vcap port and it worked (I can connect to the app url
from curl and chrome)
then I moved this single file into my modules.jar (the real app)
and configure it as main class, but here it doesn't work.

here is the class in question:
public class HelloHandler extends AbstractHandler
{
public static void main(String[] args)
{
int VCAP_APP_PORT =
Integer.parseInt((System.getenv("VCAP_APP_PORT") != null ?
System.getenv("VCAP_APP_PORT") : "8080"));

Server server = new Server(VCAP_APP_PORT);
server.setHandler(new HelloHandler());

try
{
server.start();
server.join();
}
catch (Exception exception)
{
System.out.println("Failed to start embedded jetty
server.");
exception.printStackTrace();
}
}

final String greeting;
final String body;

public HelloHandler()
{
this("Hello World");
}

public HelloHandler( String greeting )
{
this(greeting, null);
}

public HelloHandler( String greeting, String body )
{
this.greeting = greeting;
this.body = body;
}

public void handle( String target,
Request baseRequest,
HttpServletRequest request,
HttpServletResponse response ) throws
IOException,

ServletException
{
response.setContentType("text/html; charset=utf-8");
response.setStatus(HttpServletResponse.SC_OK);

PrintWriter out = response.getWriter();

out.println("<h1>" + greeting + "</h1>");
out.println("<p>Current directory: " +
getCurrentDirectory() + "</p>");
out.println("<p> Parent directory: " + getParentDirectory()
+ "</p>");
for(File file: getFiles()) {
out.println("<p>" + file.getAbsolutePath() + "</p>");
}

if (body != null)
{
out.println(body);
}

baseRequest.setHandled(true);
}
private List<File> getFiles() {
return Arrays.asList(new File(".").listFiles());
}

private String getCurrentDirectory()
{
String directory = null;
try
{
directory = new File(".").getCanonicalPath();
}
catch (Exception e)
{
e.printStackTrace();
}
return directory;
}

private String getParentDirectory()
{
String directory = null;
try
{
directory = new File("..").getCanonicalPath();
}
catch (Exception e)
{
e.printStackTrace();
}
return directory;
}
}


On Tue, Jun 9, 2015 at 5:34 PM, Daniel Mikusa <dmikusa(a)pivotal.io
wrote:
Double check that your app is configured to log to STDOUT /
STDER. If it's logging to a file, you won't see the messages since the
system only captures STDOUT & STDERR.

Also, you might try a demo app like spring-music, just to make
sure your install is working OK.

https://github.com/cloudfoundry-samples/spring-music

Dan

On Tue, Jun 9, 2015 at 10:40 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Same logs I still can't see what went wrong!

On Tue, Jun 9, 2015 at 3:58 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

And so you'd want to set `command` to `sleep 2 &&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/
bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m.. -memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java
-cp $PWD/. -Djava.io.tmpdir=$TMPDIR -XX:OnOutOfMemoryError=$PWD/.
java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler`.

Dan

On Tue, Jun 9, 2015 at 9:29 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Here is the content of *detected_start_command*:
$ CF_TRACE=true cf app modules | grep "detected_start_command"
"detected_start_command":
"CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler",

{"guid":"2dabf3c6-1736-4eb4-9bb7-40dc58bce246","name":"modules","routes":[{"guid":"014f77ed-e450-4209-be23-40e4a8257cfc","host":"modules-rhythmic-uvulatomy","domain":{"guid":"a77afa99-76d6-4f3f-9e07-27775597709f","name":"
10.244.0.34.xip.io
"}}],"running_instances":0,"services":[],"available_domains":[{"guid":"a77afa99-76d6-4f3f-9e07-27775597709f","name":"
10.244.0.34.xip.io
"}],"name":"modules","production":false,"space_guid":"381707f7-88d4-4f6e-bd7c-80d7f0699b0f","stack_guid":"3431865a-f165-4e75-9221-4f418e9de889","buildpack":"
https://github.com/cloudfoundry/java-buildpack","detected_buildpack":null,"environment_json":{"JAVA_OPTS":"$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000"},"memory":4096,"instances":1,"disk_quota":2048,"state":"STARTED","version":"e8f0e18e-7c09-4143-a17c-83a6e32eed84","command":null,"console":false,"debug":null,"staging_task_id":"1249276465c64c1884d401452d2365af","package_state":"STAGED","health_check_type":"port","health_check_timeout":180,"staging_failed_reason":null,"diego":false,"docker_image":null,"package_updated_at":"2015-06-08T07:56:44Z","detected_start_command":"CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler"}


On Tue, Jun 9, 2015 at 1:35 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

On Tue, Jun 9, 2015 at 5:05 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Hi, thanks for the hint, I tried to add to the manifest the
following entry:
command: sleep 2 && java -jar modules.jar
The Java build pack does not put Java on the path, which is
what you're seeing in the error below. I will usually run `CF_TRACE=true
cf app <app-name> | grep "detected_start_command"` which shows the command
the build pack detected and then just copy & paste that into my custom
command.



but it looks like it's not appropriate as I see in the logs:
2015-06-08T05:34:03.71+0200 [App/0] ERR bash: java:
command not found

On Mon, Jun 8, 2015 at 6:32 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

On Mon, Jun 8, 2015 at 12:03 PM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

I'm trying to push an executable JAR to a bosh-lite
instance, on the logs I see CF trying many times to restart the app and
eventually fail.

I can't find out why the app crashes as there is no app
logs returned by CF. I tried to remotely debug the app (as described in
[1]) but nothing happens on eclipse. Any hint to solve this issue is
appreciated.
There's a known issue that occurs when an app starts and
fails in rapid succession and results in the log entries being missed. If
you add a couple second pause into the app, it will give the system enough
time to attach the logger and you'll see the output generated by the
crashing app.

You can do this with a custom start command `cf push -c
'sleep 2 && <normal-cmd>'` or with a `.profile.d` script that sleeps for a
couple seconds.

Dan




1. I'm launching the app with:
*cf push -t 180*

2. Here is the manifest.yml content:
---
applications:
- name: modules
memory: 4G
instances: 1
host: modules-${random-word}
path: modules.jar
buildpack:
https://github.com/cloudfoundry/java-buildpack
env:
JAVA_OPTS: "$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000"

3. Here is the log:
Connecté, le dumping journaux récents pour application
modules en org heavenize / espace dev comme admin...

2015-06-07T12:05:15.69+0200 [API] OUT Created app
with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:05:16.10+0200 [API] OUT Updated app
with guid 1fbf3378-6512-46de-bae4-02ee30275464
({"route"=>"68e27d8d-4ff6-443b-a3e0-416c40d325d3"})
2015-06-07T12:12:21.55+0200 [DEA] OUT Got staging
request for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:12:25.40+0200 [API] OUT Updated app
with guid 1fbf3378-6512-46de-bae4-02ee30275464 ({"state"=>"STARTED"})
2015-06-07T12:12:38.08+0200 [STG] OUT ----->
Downloaded app package (163M)
2015-06-07T12:13:40.93+0200 [STG] ERR Cloning
into '/tmp/buildpacks/java-buildpack'...
2015-06-07T12:14:01.67+0200 [STG] OUT -----> Java
Buildpack Version: c862ac8 |
https://github.com/cloudfoundry/java-buildpack#c862ac8
2015-06-07T12:14:13.89+0200 [STG] OUT ----->
Downloading Open Jdk JRE 1.8.0_45 from
https://download.run.pivotal.io/openjdk/lucid/x86_64/openjdk-1.8.0_45.tar.gz
(11.6s)
2015-06-07T12:14:15.34+0200 [STG] OUT
Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.4s)
2015-06-07T12:14:15.89+0200 [STG] OUT ----->
Downloading Open JDK Like Memory Calculator 1.1.1_RELEASE from
https://download.run.pivotal.io/memory-calculator/lucid/x86_64/memory-calculator-1.1.1_RELEASE
(0.5s)
2015-06-07T12:14:15.90+0200 [STG] OUT Memory
Settings: -XX:MaxMetaspaceSize=419430K -XX:MetaspaceSize=419430K -Xss1M
-Xmx3G -Xms3G
2015-06-07T12:15:43.44+0200 [STG] OUT ----->
Uploading droplet (151M)
2015-06-07T12:16:07.51+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:16:29.66+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"9d55c5f791324d358bffb4c961a4c7ee", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672189}
2015-06-07T12:17:14.18+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:31.10+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"2ae0c26f33864f40989ee870a1b9e3db", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672251}
2015-06-07T12:17:38.48+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:38.48+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:38.48+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:55.31+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:11.82+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"dc872d38f3324af481c82ba67f0e216c", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672291}
2015-06-07T12:18:18.69+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:18.69+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:18.69+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:34.06+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:50.98+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"623c3af7e3e84801b6fd44eeee9c0a12", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672330}
2015-06-07T12:18:58.80+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:58.80+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:58.80+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:34.08+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:50.36+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"40727eea293146948af197e13443843c", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672390}
2015-06-07T12:19:59.01+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:59.01+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:59.01+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:04.12+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:20.61+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"f7ffff55692a418c847f4f37be574ddf", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672480}
2015-06-07T12:21:29.43+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:29.43+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:29.47+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:34.16+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:49.97+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"4581e97c6b0f4504b8d64a5c69d6787b", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672629}
2015-06-07T12:23:50.29+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:50.29+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:50.29+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:14.24+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:29.82+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"f98749490a6743598f57d3848eb06177", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672909}
2015-06-07T12:28:31.73+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:31.73+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:31.73+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464



[1]
http://docs.cloudfoundry.org/buildpacks/java/java-tips.html#debugging



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: R: Re: Monitor all outbound connections from apps in warden

CF Runtime
 

I would expect this to be possible. The easiest thing would probably be to
write a shell script that both runs your application, and also starts a
script that does the netstat output. Then set that script to run as your
apps custom start command "cf push my_app -c run_app_and_log.sh".

Joseph Palermo
CF Runtime Team

On Mon, Jun 1, 2015 at 12:07 AM, Michael Grifalconi <
michael.grifalconi(a)studenti.unimi.it> wrote:

Hello, thank you for the hint!

I'd prefer to do something at application level, like a shell script to
run in parallel to the application that every X seconds prints the output
of netstat, as the standard output is taken as a log on CF apps. Is it
possible?

(I'm really sorry and embarrassed about the spam after my email signature,
this is due to my University and I can't avoid it :/ )

Thank you,
Michael


Il 29/05/15 20:06, *Dieu Cao * <dcao(a)pivotal.io> ha scritto:

You could set up a security group that logs all outbound connections.
These are logged on the DEAs.
You would then need to correlate the warden handle with the application.

I'm working with the docs team on getting this feature properly documented.

Relevant stories where this feature was added.
[1] https://www.pivotaltracker.com/story/show/73905126
[2] https://www.pivotaltracker.com/story/show/90078842

I don't know how you would do this via buildpacks.

-Dieu
CF Runtime PM


On Fri, May 29, 2015 at 6:59 AM, Michael Grifalconi <
michael.grifalconi(a)studenti.unimi.it> wrote:

Hello all,

How can I monitor (and log) all the outbound connection made from an
application?

I would like to do by editing buildpacks:

edit the buildpack to run a netstat command every 10 sec and send a log
of the estabilished connections..


I would also be able to sniff the traffic, is it possible to run a
tcpdump with some filters and send logs with the result? All by editing the
buildpack. I think the process will not have the necessary privileges..


Any hint is appreciated!

Thank you!

Michael
------------------------------

[image: 5xmilleUniMi]
<http://www.unimi.it/13084.htm?utm_source=firmaMail&utm_medium=email&utm_content=linkFirmaEmail&utm_campaign=5xmille>


* La tua firma per la sua idea. Per tutti noi Per destinare il 5x1000
all'Università degli Studi di Milano: indicare nella dichiarazione dei
redditi il codice fiscale 80012650158. *
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

------------------------------

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


(No subject)

CF Runtime
 

You'll need to increase cc.maximum_app_disk_in_mb in your bosh deployment
manifest. That defines the maximum disk an app is allowed to request.

Joseph Palermo
CF Runtime Team

On Fri, Jun 12, 2015 at 1:31 AM, Arbi Akhina <arbi.akhina(a)gmail.com> wrote:

I'm trying to increase the disk quota in the manifest to 10G to try deploy
the original big jar and I'm seeing the following error:
E:\src\samples\cloudfoundry\modules>cf push
En utilisant le fichier manifeste
E:\src\samples\cloudfoundry\modules\manifest.yml

Création d'application modules en org heavenize / espace dev comme admin...
RATÉ
Erreur du serveur, code d'état: 400, code erreur: 100001, message: The app
is invalid: disk_quota too much disk requested (must be less than 2048)

On Fri, Jun 12, 2015 at 10:21 AM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

Yeah, I get Jetty logs as below:
2015-06-11T04:27:46.34+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:27:46
+0000] "GET / HTTP/1.1" 200 1098 "-" "curl/7.29.0" 10.0.2.15:58442
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:a94d78ae-81a8-4ca4-55f9-9ce8fe9d8f7b
response_time:0.027328613 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f
2015-06-11T04:42:23.03+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:42:23
+0000] "GET / HTTP/1.1" 200 1098 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Dragon/42.2.2.138
Chrome/42.0.2311.135 Safari/537.36" 10.0.2.15:32833
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:e24de0c2-d81c-4cd6-76cc-7077bb5ec26c
response_time:0.024409247 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f
2015-06-11T04:42:23.29+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:42:23
+0000] "GET /favicon.ico HTTP/1.1" 200 1098 "
http://modules-popeyed-nonascetic.10.244.0.34.xip.io/" "Mozilla/5.0
(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Dragon/
42.2.2.138 Chrome/42.0.2311.135 Safari/537.36" 10.0.2.15:32833
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:b17c21b4-eba7-4fd5-4fac-4826f89feaae
response_time:0.023473933 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f


On Thu, Jun 11, 2015 at 5:48 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Thu, Jun 11, 2015 at 11:37 AM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

I removed static assets as well as many classes that were not used,
I left classes related to jetty, javax (managment, ..), apache commons
io, and the package (with all classes) where the main class is located.
Interesting. Do you get log output from the application when it's
working? I'm curious because you weren't seeing this when it was failing.
Trying to understand if the app is not logging properly or if your logs
were being lost somehow.



I didn't try on public cfs like PWS (what is this one?) and BlueMix.
Pivotal Web Services. Similar to Bluemix, it's a public CF that you can
sign up to use. There are others too. The idea here is just to test this
on a larger, more powerful system and see if the problems go away.

Dan




On Thu, Jun 11, 2015 at 4:42 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Thu, Jun 11, 2015 at 10:31 AM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

Hi, I had to reduce the size of the jar to ~ 51M to get the simple
class that runs jetty to work on bosh-lite!!!
Interesting. Out of curiosity what did you remove to make it
smaller? Static assets? Also have you tried a larger CF install like PWS
/ BlueMix, just to see if this is an issue specific to bosh-lite.


Is there any kind of size restriction for executable jars??
The system can put a limit on how big of an application you can
upload. I don't know about bosh-lite specifically, but the default is 1G.
Also, when you exceed that you'll get a message from the cf cli saying your
app is too big. It won't silently fail, which seems to be what happened
here.

Dan



Here is the system information of the warden container on which the
jar is running:
Java version : 1.8.0_45-
Java Spec version : 1.8
JVM version : 25.45-b02
JVM vendor : Oracle Corporation
JVM name : OpenJDK 64-Bit Server VM
Os Name : Linux
Os Architecture : amd64
Os version : 3.13.0-44-generic
Number of processors : 4
Max memory : 3087007744
Total memory : 3087007744
Available memory : 2958157616
Free work disk space : 17898352640
VM Args : [-Djava.io.tmpdir=/home/vcap/tmp,
-XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh,
-Xss1M, -Xmx3G, -Xms3G, -XX:MaxMetaspaceSize=419430K,
-XX:MetaspaceSize=419430K]

On Tue, Jun 9, 2015 at 7:01 PM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

no I meant the source is for the company I work for, not my personal
project.

On Tue, Jun 9, 2015 at 6:58 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Tue, Jun 9, 2015 at 12:43 PM, Arbi Akhina <arbi.akhina(a)gmail.com
wrote:
I don't own the source, but here is the jar file
http://tempsend.com/6D745B7B07/7033/modules.jar
you can try to run it locally with java -jar modules.jar then
connect to localhost:8080
Hmm, OK. I thought you had a minimal example from you last note.
I'm not really up for running untrusted binary code. If you can put a
minimal code sample together, I'll give it a try.



An here is the manifest.yml i'm using:
---
applications:
- name: modules
memory: 4G
disk_quota: 2G
timeout: 180
instances: 1
host: modules-${random-word}
path: modules.jar
buildpack: https://github.com/cloudfoundry/java-buildpack
command: sleep 2 &&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY com.heavenize.osmoze.kernel.HelloHandler
# env:
# JAVA_OPTS: "$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000"

I'm having both files in the same folder modules/ and I push with cf
push
That seems OK. Is this app designed to run on CF? By that, I mean
is it configure to log to STDOUT / STDERR and is it going to listen on the
right port (i.e $PORT)?

Dan




Thanks a lot

On Tue, Jun 9, 2015 at 6:25 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

I'd be happy to try the demo and see if it works for me. Can you
post a complete project on github so I can just `git clone` & build?

Dan

On Tue, Jun 9, 2015 at 12:21 PM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

I've tried with a simple app that just launches an embedded
jetty that listen on the vcap port and it worked (I can connect to the app
url from curl and chrome)
then I moved this single file into my modules.jar (the real app)
and configure it as main class, but here it doesn't work.

here is the class in question:
public class HelloHandler extends AbstractHandler
{
public static void main(String[] args)
{
int VCAP_APP_PORT =
Integer.parseInt((System.getenv("VCAP_APP_PORT") != null ?
System.getenv("VCAP_APP_PORT") : "8080"));

Server server = new Server(VCAP_APP_PORT);
server.setHandler(new HelloHandler());

try
{
server.start();
server.join();
}
catch (Exception exception)
{
System.out.println("Failed to start embedded jetty
server.");
exception.printStackTrace();
}
}

final String greeting;
final String body;

public HelloHandler()
{
this("Hello World");
}

public HelloHandler( String greeting )
{
this(greeting, null);
}

public HelloHandler( String greeting, String body )
{
this.greeting = greeting;
this.body = body;
}

public void handle( String target,
Request baseRequest,
HttpServletRequest request,
HttpServletResponse response ) throws
IOException,

ServletException
{
response.setContentType("text/html; charset=utf-8");
response.setStatus(HttpServletResponse.SC_OK);

PrintWriter out = response.getWriter();

out.println("<h1>" + greeting + "</h1>");
out.println("<p>Current directory: " +
getCurrentDirectory() + "</p>");
out.println("<p> Parent directory: " +
getParentDirectory() + "</p>");
for(File file: getFiles()) {
out.println("<p>" + file.getAbsolutePath() + "</p>");
}

if (body != null)
{
out.println(body);
}

baseRequest.setHandled(true);
}
private List<File> getFiles() {
return Arrays.asList(new File(".").listFiles());
}

private String getCurrentDirectory()
{
String directory = null;
try
{
directory = new File(".").getCanonicalPath();
}
catch (Exception e)
{
e.printStackTrace();
}
return directory;
}

private String getParentDirectory()
{
String directory = null;
try
{
directory = new File("..").getCanonicalPath();
}
catch (Exception e)
{
e.printStackTrace();
}
return directory;
}
}


On Tue, Jun 9, 2015 at 5:34 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

Double check that your app is configured to log to STDOUT /
STDER. If it's logging to a file, you won't see the messages since the
system only captures STDOUT & STDERR.

Also, you might try a demo app like spring-music, just to make
sure your install is working OK.

https://github.com/cloudfoundry-samples/spring-music

Dan

On Tue, Jun 9, 2015 at 10:40 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Same logs I still can't see what went wrong!

On Tue, Jun 9, 2015 at 3:58 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

And so you'd want to set `command` to `sleep 2 &&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/
bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m.. -memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java
-cp $PWD/. -Djava.io.tmpdir=$TMPDIR -XX:OnOutOfMemoryError=$PWD/.
java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_
socket,address=192.168.2.8:8000 com.heavenize.osmoze.kernel.
HelloHandler`.

Dan

On Tue, Jun 9, 2015 at 9:29 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Here is the content of *detected_start_command*:
$ CF_TRACE=true cf app modules | grep
"detected_start_command"
"detected_start_command":
"CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler",

{"guid":"2dabf3c6-1736-4eb4-9bb7-40dc58bce246","name":"modules","routes":[{"guid":"014f77ed-e450-4209-be23-40e4a8257cfc","host":"modules-rhythmic-uvulatomy","domain":{"guid":"a77afa99-76d6-4f3f-9e07-27775597709f","name":"
10.244.0.34.xip.io
"}}],"running_instances":0,"services":[],"available_domains":[{"guid":"a77afa99-76d6-4f3f-9e07-27775597709f","name":"
10.244.0.34.xip.io
"}],"name":"modules","production":false,"space_guid":"381707f7-88d4-4f6e-bd7c-80d7f0699b0f","stack_guid":"3431865a-f165-4e75-9221-4f418e9de889","buildpack":"
https://github.com/cloudfoundry/java-buildpack","detected_buildpack":null,"environment_json":{"JAVA_OPTS":"$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000"},"memory":4096,"instances":1,"disk_quota":2048,"state":"STARTED","version":"e8f0e18e-7c09-4143-a17c-83a6e32eed84","command":null,"console":false,"debug":null,"staging_task_id":"1249276465c64c1884d401452d2365af","package_state":"STAGED","health_check_type":"port","health_check_timeout":180,"staging_failed_reason":null,"diego":false,"docker_image":null,"package_updated_at":"2015-06-08T07:56:44Z","detected_start_command":"CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler"}


On Tue, Jun 9, 2015 at 1:35 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

On Tue, Jun 9, 2015 at 5:05 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Hi, thanks for the hint, I tried to add to the manifest
the following entry:
command: sleep 2 && java -jar modules.jar
The Java build pack does not put Java on the path, which is
what you're seeing in the error below. I will usually run `CF_TRACE=true
cf app <app-name> | grep "detected_start_command"` which shows the command
the build pack detected and then just copy & paste that into my custom
command.



but it looks like it's not appropriate as I see in the
logs:
2015-06-08T05:34:03.71+0200 [App/0] ERR bash: java:
command not found

On Mon, Jun 8, 2015 at 6:32 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

On Mon, Jun 8, 2015 at 12:03 PM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

I'm trying to push an executable JAR to a bosh-lite
instance, on the logs I see CF trying many times to restart the app and
eventually fail.

I can't find out why the app crashes as there is no app
logs returned by CF. I tried to remotely debug the app (as described in
[1]) but nothing happens on eclipse. Any hint to solve this issue is
appreciated.
There's a known issue that occurs when an app starts and
fails in rapid succession and results in the log entries being missed. If
you add a couple second pause into the app, it will give the system enough
time to attach the logger and you'll see the output generated by the
crashing app.

You can do this with a custom start command `cf push -c
'sleep 2 && <normal-cmd>'` or with a `.profile.d` script that sleeps for a
couple seconds.

Dan




1. I'm launching the app with:
*cf push -t 180*

2. Here is the manifest.yml content:
---
applications:
- name: modules
memory: 4G
instances: 1
host: modules-${random-word}
path: modules.jar
buildpack:
https://github.com/cloudfoundry/java-buildpack
env:
JAVA_OPTS: "$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000"

3. Here is the log:
Connecté, le dumping journaux récents pour application
modules en org heavenize / espace dev comme admin...

2015-06-07T12:05:15.69+0200 [API] OUT Created app
with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:05:16.10+0200 [API] OUT Updated app
with guid 1fbf3378-6512-46de-bae4-02ee30275464
({"route"=>"68e27d8d-4ff6-443b-a3e0-416c40d325d3"})
2015-06-07T12:12:21.55+0200 [DEA] OUT Got staging
request for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:12:25.40+0200 [API] OUT Updated app
with guid 1fbf3378-6512-46de-bae4-02ee30275464 ({"state"=>"STARTED"})
2015-06-07T12:12:38.08+0200 [STG] OUT ----->
Downloaded app package (163M)
2015-06-07T12:13:40.93+0200 [STG] ERR Cloning
into '/tmp/buildpacks/java-buildpack'...
2015-06-07T12:14:01.67+0200 [STG] OUT ----->
Java Buildpack Version: c862ac8 |
https://github.com/cloudfoundry/java-buildpack#c862ac8
2015-06-07T12:14:13.89+0200 [STG] OUT ----->
Downloading Open Jdk JRE 1.8.0_45 from
https://download.run.pivotal.io/openjdk/lucid/x86_64/openjdk-1.8.0_45.tar.gz
(11.6s)
2015-06-07T12:14:15.34+0200 [STG] OUT
Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.4s)
2015-06-07T12:14:15.89+0200 [STG] OUT ----->
Downloading Open JDK Like Memory Calculator 1.1.1_RELEASE from
https://download.run.pivotal.io/memory-calculator/lucid/x86_64/memory-calculator-1.1.1_RELEASE
(0.5s)
2015-06-07T12:14:15.90+0200 [STG] OUT Memory
Settings: -XX:MaxMetaspaceSize=419430K -XX:MetaspaceSize=419430K -Xss1M
-Xmx3G -Xms3G
2015-06-07T12:15:43.44+0200 [STG] OUT ----->
Uploading droplet (151M)
2015-06-07T12:16:07.51+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:16:29.66+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"9d55c5f791324d358bffb4c961a4c7ee", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672189}
2015-06-07T12:17:14.18+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:31.10+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"2ae0c26f33864f40989ee870a1b9e3db", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672251}
2015-06-07T12:17:38.48+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:38.48+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:38.48+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:55.31+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:11.82+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"dc872d38f3324af481c82ba67f0e216c", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672291}
2015-06-07T12:18:18.69+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:18.69+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:18.69+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:34.06+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:50.98+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"623c3af7e3e84801b6fd44eeee9c0a12", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672330}
2015-06-07T12:18:58.80+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:58.80+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:58.80+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:34.08+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:50.36+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"40727eea293146948af197e13443843c", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672390}
2015-06-07T12:19:59.01+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:59.01+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:59.01+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:04.12+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:20.61+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"f7ffff55692a418c847f4f37be574ddf", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672480}
2015-06-07T12:21:29.43+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:29.43+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:29.47+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:34.16+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:49.97+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"4581e97c6b0f4504b8d64a5c69d6787b", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672629}
2015-06-07T12:23:50.29+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:50.29+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:50.29+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:14.24+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:29.82+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"f98749490a6743598f57d3848eb06177", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672909}
2015-06-07T12:28:31.73+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:31.73+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:31.73+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464



[1]
http://docs.cloudfoundry.org/buildpacks/java/java-tips.html#debugging



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


(No subject)

Arbi Akhina
 

I'm trying to increase the disk quota in the manifest to 10G to try deploy
the original big jar and I'm seeing the following error:
E:\src\samples\cloudfoundry\modules>cf push
En utilisant le fichier manifeste
E:\src\samples\cloudfoundry\modules\manifest.yml

Création d'application modules en org heavenize / espace dev comme admin...
RATÉ
Erreur du serveur, code d'état: 400, code erreur: 100001, message: The app
is invalid: disk_quota too much disk requested (must be less than 2048)

On Fri, Jun 12, 2015 at 10:21 AM, Arbi Akhina <arbi.akhina(a)gmail.com> wrote:

Yeah, I get Jetty logs as below:
2015-06-11T04:27:46.34+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:27:46
+0000] "GET / HTTP/1.1" 200 1098 "-" "curl/7.29.0" 10.0.2.15:58442
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:a94d78ae-81a8-4ca4-55f9-9ce8fe9d8f7b
response_time:0.027328613 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f
2015-06-11T04:42:23.03+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:42:23
+0000] "GET / HTTP/1.1" 200 1098 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Dragon/42.2.2.138
Chrome/42.0.2311.135 Safari/537.36" 10.0.2.15:32833
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:e24de0c2-d81c-4cd6-76cc-7077bb5ec26c
response_time:0.024409247 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f
2015-06-11T04:42:23.29+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:42:23
+0000] "GET /favicon.ico HTTP/1.1" 200 1098 "
http://modules-popeyed-nonascetic.10.244.0.34.xip.io/" "Mozilla/5.0
(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Dragon/
42.2.2.138 Chrome/42.0.2311.135 Safari/537.36" 10.0.2.15:32833
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:b17c21b4-eba7-4fd5-4fac-4826f89feaae
response_time:0.023473933 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f


On Thu, Jun 11, 2015 at 5:48 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

On Thu, Jun 11, 2015 at 11:37 AM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

I removed static assets as well as many classes that were not used,
I left classes related to jetty, javax (managment, ..), apache commons
io, and the package (with all classes) where the main class is located.
Interesting. Do you get log output from the application when it's
working? I'm curious because you weren't seeing this when it was failing.
Trying to understand if the app is not logging properly or if your logs
were being lost somehow.



I didn't try on public cfs like PWS (what is this one?) and BlueMix.
Pivotal Web Services. Similar to Bluemix, it's a public CF that you can
sign up to use. There are others too. The idea here is just to test this
on a larger, more powerful system and see if the problems go away.

Dan




On Thu, Jun 11, 2015 at 4:42 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Thu, Jun 11, 2015 at 10:31 AM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

Hi, I had to reduce the size of the jar to ~ 51M to get the simple
class that runs jetty to work on bosh-lite!!!
Interesting. Out of curiosity what did you remove to make it smaller?
Static assets? Also have you tried a larger CF install like PWS / BlueMix,
just to see if this is an issue specific to bosh-lite.


Is there any kind of size restriction for executable jars??
The system can put a limit on how big of an application you can
upload. I don't know about bosh-lite specifically, but the default is 1G.
Also, when you exceed that you'll get a message from the cf cli saying your
app is too big. It won't silently fail, which seems to be what happened
here.

Dan



Here is the system information of the warden container on which the
jar is running:
Java version : 1.8.0_45-
Java Spec version : 1.8
JVM version : 25.45-b02
JVM vendor : Oracle Corporation
JVM name : OpenJDK 64-Bit Server VM
Os Name : Linux
Os Architecture : amd64
Os version : 3.13.0-44-generic
Number of processors : 4
Max memory : 3087007744
Total memory : 3087007744
Available memory : 2958157616
Free work disk space : 17898352640
VM Args : [-Djava.io.tmpdir=/home/vcap/tmp,
-XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh,
-Xss1M, -Xmx3G, -Xms3G, -XX:MaxMetaspaceSize=419430K,
-XX:MetaspaceSize=419430K]

On Tue, Jun 9, 2015 at 7:01 PM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

no I meant the source is for the company I work for, not my personal
project.

On Tue, Jun 9, 2015 at 6:58 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Tue, Jun 9, 2015 at 12:43 PM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

I don't own the source, but here is the jar file
http://tempsend.com/6D745B7B07/7033/modules.jar
you can try to run it locally with java -jar modules.jar then
connect to localhost:8080
Hmm, OK. I thought you had a minimal example from you last note.
I'm not really up for running untrusted binary code. If you can put a
minimal code sample together, I'll give it a try.



An here is the manifest.yml i'm using:
---
applications:
- name: modules
memory: 4G
disk_quota: 2G
timeout: 180
instances: 1
host: modules-${random-word}
path: modules.jar
buildpack: https://github.com/cloudfoundry/java-buildpack
command: sleep 2 &&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY com.heavenize.osmoze.kernel.HelloHandler
# env:
# JAVA_OPTS: "$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000"

I'm having both files in the same folder modules/ and I push with cf
push
That seems OK. Is this app designed to run on CF? By that, I mean
is it configure to log to STDOUT / STDERR and is it going to listen on the
right port (i.e $PORT)?

Dan




Thanks a lot

On Tue, Jun 9, 2015 at 6:25 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

I'd be happy to try the demo and see if it works for me. Can you
post a complete project on github so I can just `git clone` & build?

Dan

On Tue, Jun 9, 2015 at 12:21 PM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

I've tried with a simple app that just launches an embedded jetty
that listen on the vcap port and it worked (I can connect to the app url
from curl and chrome)
then I moved this single file into my modules.jar (the real app)
and configure it as main class, but here it doesn't work.

here is the class in question:
public class HelloHandler extends AbstractHandler
{
public static void main(String[] args)
{
int VCAP_APP_PORT =
Integer.parseInt((System.getenv("VCAP_APP_PORT") != null ?
System.getenv("VCAP_APP_PORT") : "8080"));

Server server = new Server(VCAP_APP_PORT);
server.setHandler(new HelloHandler());

try
{
server.start();
server.join();
}
catch (Exception exception)
{
System.out.println("Failed to start embedded jetty
server.");
exception.printStackTrace();
}
}

final String greeting;
final String body;

public HelloHandler()
{
this("Hello World");
}

public HelloHandler( String greeting )
{
this(greeting, null);
}

public HelloHandler( String greeting, String body )
{
this.greeting = greeting;
this.body = body;
}

public void handle( String target,
Request baseRequest,
HttpServletRequest request,
HttpServletResponse response ) throws
IOException,

ServletException
{
response.setContentType("text/html; charset=utf-8");
response.setStatus(HttpServletResponse.SC_OK);

PrintWriter out = response.getWriter();

out.println("<h1>" + greeting + "</h1>");
out.println("<p>Current directory: " +
getCurrentDirectory() + "</p>");
out.println("<p> Parent directory: " + getParentDirectory()
+ "</p>");
for(File file: getFiles()) {
out.println("<p>" + file.getAbsolutePath() + "</p>");
}

if (body != null)
{
out.println(body);
}

baseRequest.setHandled(true);
}
private List<File> getFiles() {
return Arrays.asList(new File(".").listFiles());
}

private String getCurrentDirectory()
{
String directory = null;
try
{
directory = new File(".").getCanonicalPath();
}
catch (Exception e)
{
e.printStackTrace();
}
return directory;
}

private String getParentDirectory()
{
String directory = null;
try
{
directory = new File("..").getCanonicalPath();
}
catch (Exception e)
{
e.printStackTrace();
}
return directory;
}
}


On Tue, Jun 9, 2015 at 5:34 PM, Daniel Mikusa <dmikusa(a)pivotal.io
wrote:
Double check that your app is configured to log to STDOUT /
STDER. If it's logging to a file, you won't see the messages since the
system only captures STDOUT & STDERR.

Also, you might try a demo app like spring-music, just to make
sure your install is working OK.

https://github.com/cloudfoundry-samples/spring-music

Dan

On Tue, Jun 9, 2015 at 10:40 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Same logs I still can't see what went wrong!

On Tue, Jun 9, 2015 at 3:58 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

And so you'd want to set `command` to `sleep 2 &&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/
bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m.. -memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java
-cp $PWD/. -Djava.io.tmpdir=$TMPDIR -XX:OnOutOfMemoryError=$PWD/.
java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler`.

Dan

On Tue, Jun 9, 2015 at 9:29 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Here is the content of *detected_start_command*:
$ CF_TRACE=true cf app modules | grep "detected_start_command"
"detected_start_command":
"CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler",

{"guid":"2dabf3c6-1736-4eb4-9bb7-40dc58bce246","name":"modules","routes":[{"guid":"014f77ed-e450-4209-be23-40e4a8257cfc","host":"modules-rhythmic-uvulatomy","domain":{"guid":"a77afa99-76d6-4f3f-9e07-27775597709f","name":"
10.244.0.34.xip.io
"}}],"running_instances":0,"services":[],"available_domains":[{"guid":"a77afa99-76d6-4f3f-9e07-27775597709f","name":"
10.244.0.34.xip.io
"}],"name":"modules","production":false,"space_guid":"381707f7-88d4-4f6e-bd7c-80d7f0699b0f","stack_guid":"3431865a-f165-4e75-9221-4f418e9de889","buildpack":"
https://github.com/cloudfoundry/java-buildpack","detected_buildpack":null,"environment_json":{"JAVA_OPTS":"$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000"},"memory":4096,"instances":1,"disk_quota":2048,"state":"STARTED","version":"e8f0e18e-7c09-4143-a17c-83a6e32eed84","command":null,"console":false,"debug":null,"staging_task_id":"1249276465c64c1884d401452d2365af","package_state":"STAGED","health_check_type":"port","health_check_timeout":180,"staging_failed_reason":null,"diego":false,"docker_image":null,"package_updated_at":"2015-06-08T07:56:44Z","detected_start_command":"CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler"}


On Tue, Jun 9, 2015 at 1:35 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

On Tue, Jun 9, 2015 at 5:05 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Hi, thanks for the hint, I tried to add to the manifest the
following entry:
command: sleep 2 && java -jar modules.jar
The Java build pack does not put Java on the path, which is
what you're seeing in the error below. I will usually run `CF_TRACE=true
cf app <app-name> | grep "detected_start_command"` which shows the command
the build pack detected and then just copy & paste that into my custom
command.



but it looks like it's not appropriate as I see in the logs:
2015-06-08T05:34:03.71+0200 [App/0] ERR bash: java:
command not found

On Mon, Jun 8, 2015 at 6:32 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

On Mon, Jun 8, 2015 at 12:03 PM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

I'm trying to push an executable JAR to a bosh-lite
instance, on the logs I see CF trying many times to restart the app and
eventually fail.

I can't find out why the app crashes as there is no app
logs returned by CF. I tried to remotely debug the app (as described in
[1]) but nothing happens on eclipse. Any hint to solve this issue is
appreciated.
There's a known issue that occurs when an app starts and
fails in rapid succession and results in the log entries being missed. If
you add a couple second pause into the app, it will give the system enough
time to attach the logger and you'll see the output generated by the
crashing app.

You can do this with a custom start command `cf push -c
'sleep 2 && <normal-cmd>'` or with a `.profile.d` script that sleeps for a
couple seconds.

Dan




1. I'm launching the app with:
*cf push -t 180*

2. Here is the manifest.yml content:
---
applications:
- name: modules
memory: 4G
instances: 1
host: modules-${random-word}
path: modules.jar
buildpack:
https://github.com/cloudfoundry/java-buildpack
env:
JAVA_OPTS: "$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000"

3. Here is the log:
Connecté, le dumping journaux récents pour application
modules en org heavenize / espace dev comme admin...

2015-06-07T12:05:15.69+0200 [API] OUT Created app
with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:05:16.10+0200 [API] OUT Updated app
with guid 1fbf3378-6512-46de-bae4-02ee30275464
({"route"=>"68e27d8d-4ff6-443b-a3e0-416c40d325d3"})
2015-06-07T12:12:21.55+0200 [DEA] OUT Got staging
request for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:12:25.40+0200 [API] OUT Updated app
with guid 1fbf3378-6512-46de-bae4-02ee30275464 ({"state"=>"STARTED"})
2015-06-07T12:12:38.08+0200 [STG] OUT ----->
Downloaded app package (163M)
2015-06-07T12:13:40.93+0200 [STG] ERR Cloning
into '/tmp/buildpacks/java-buildpack'...
2015-06-07T12:14:01.67+0200 [STG] OUT -----> Java
Buildpack Version: c862ac8 |
https://github.com/cloudfoundry/java-buildpack#c862ac8
2015-06-07T12:14:13.89+0200 [STG] OUT ----->
Downloading Open Jdk JRE 1.8.0_45 from
https://download.run.pivotal.io/openjdk/lucid/x86_64/openjdk-1.8.0_45.tar.gz
(11.6s)
2015-06-07T12:14:15.34+0200 [STG] OUT
Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.4s)
2015-06-07T12:14:15.89+0200 [STG] OUT ----->
Downloading Open JDK Like Memory Calculator 1.1.1_RELEASE from
https://download.run.pivotal.io/memory-calculator/lucid/x86_64/memory-calculator-1.1.1_RELEASE
(0.5s)
2015-06-07T12:14:15.90+0200 [STG] OUT Memory
Settings: -XX:MaxMetaspaceSize=419430K -XX:MetaspaceSize=419430K -Xss1M
-Xmx3G -Xms3G
2015-06-07T12:15:43.44+0200 [STG] OUT ----->
Uploading droplet (151M)
2015-06-07T12:16:07.51+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:16:29.66+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"9d55c5f791324d358bffb4c961a4c7ee", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672189}
2015-06-07T12:17:14.18+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:31.10+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"2ae0c26f33864f40989ee870a1b9e3db", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672251}
2015-06-07T12:17:38.48+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:38.48+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:38.48+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:55.31+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:11.82+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"dc872d38f3324af481c82ba67f0e216c", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672291}
2015-06-07T12:18:18.69+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:18.69+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:18.69+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:34.06+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:50.98+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"623c3af7e3e84801b6fd44eeee9c0a12", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672330}
2015-06-07T12:18:58.80+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:58.80+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:58.80+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:34.08+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:50.36+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"40727eea293146948af197e13443843c", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672390}
2015-06-07T12:19:59.01+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:59.01+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:59.01+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:04.12+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:20.61+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"f7ffff55692a418c847f4f37be574ddf", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672480}
2015-06-07T12:21:29.43+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:29.43+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:29.47+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:34.16+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:49.97+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"4581e97c6b0f4504b8d64a5c69d6787b", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672629}
2015-06-07T12:23:50.29+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:50.29+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:50.29+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:14.24+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:29.82+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"f98749490a6743598f57d3848eb06177", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672909}
2015-06-07T12:28:31.73+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:31.73+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:31.73+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464



[1]
http://docs.cloudfoundry.org/buildpacks/java/java-tips.html#debugging



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


(No subject)

Arbi Akhina
 

Yeah, I get Jetty logs as below:
2015-06-11T04:27:46.34+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:27:46 +0000]
"GET / HTTP/1.1" 200 1098 "-" "curl/7.29.0" 10.0.2.15:58442
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:a94d78ae-81a8-4ca4-55f9-9ce8fe9d8f7b
response_time:0.027328613 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f
2015-06-11T04:42:23.03+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:42:23 +0000]
"GET / HTTP/1.1" 200 1098 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64)
AppleWebKit/537.36 (KHTML, like Gecko) Dragon/42.2.2.138
Chrome/42.0.2311.135 Safari/537.36" 10.0.2.15:32833
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:e24de0c2-d81c-4cd6-76cc-7077bb5ec26c
response_time:0.024409247 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f
2015-06-11T04:42:23.29+0200 [RTR] OUT
modules-popeyed-nonascetic.10.244.0.34.xip.io - [11/06/2015:02:42:23 +0000]
"GET /favicon.ico HTTP/1.1" 200 1098 "
http://modules-popeyed-nonascetic.10.244.0.34.xip.io/" "Mozilla/5.0
(Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Dragon/
42.2.2.138 Chrome/42.0.2311.135 Safari/537.36" 10.0.2.15:32833
x_forwarded_for:"192.168.2.8, 192.168.50.1, 10.0.2.15"
vcap_request_id:b17c21b4-eba7-4fd5-4fac-4826f89feaae
response_time:0.023473933 app_id:9fc4aeb6-15af-407c-baaf-9939b5417f3f

On Thu, Jun 11, 2015 at 5:48 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

On Thu, Jun 11, 2015 at 11:37 AM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

I removed static assets as well as many classes that were not used,
I left classes related to jetty, javax (managment, ..), apache commons
io, and the package (with all classes) where the main class is located.
Interesting. Do you get log output from the application when it's
working? I'm curious because you weren't seeing this when it was failing.
Trying to understand if the app is not logging properly or if your logs
were being lost somehow.



I didn't try on public cfs like PWS (what is this one?) and BlueMix.
Pivotal Web Services. Similar to Bluemix, it's a public CF that you can
sign up to use. There are others too. The idea here is just to test this
on a larger, more powerful system and see if the problems go away.

Dan




On Thu, Jun 11, 2015 at 4:42 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Thu, Jun 11, 2015 at 10:31 AM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

Hi, I had to reduce the size of the jar to ~ 51M to get the simple
class that runs jetty to work on bosh-lite!!!
Interesting. Out of curiosity what did you remove to make it smaller?
Static assets? Also have you tried a larger CF install like PWS / BlueMix,
just to see if this is an issue specific to bosh-lite.


Is there any kind of size restriction for executable jars??
The system can put a limit on how big of an application you can upload.
I don't know about bosh-lite specifically, but the default is 1G. Also,
when you exceed that you'll get a message from the cf cli saying your app
is too big. It won't silently fail, which seems to be what happened here.

Dan



Here is the system information of the warden container on which the jar
is running:
Java version : 1.8.0_45-
Java Spec version : 1.8
JVM version : 25.45-b02
JVM vendor : Oracle Corporation
JVM name : OpenJDK 64-Bit Server VM
Os Name : Linux
Os Architecture : amd64
Os version : 3.13.0-44-generic
Number of processors : 4
Max memory : 3087007744
Total memory : 3087007744
Available memory : 2958157616
Free work disk space : 17898352640
VM Args : [-Djava.io.tmpdir=/home/vcap/tmp,
-XX:OnOutOfMemoryError=/home/vcap/app/.java-buildpack/open_jdk_jre/bin/killjava.sh,
-Xss1M, -Xmx3G, -Xms3G, -XX:MaxMetaspaceSize=419430K,
-XX:MetaspaceSize=419430K]

On Tue, Jun 9, 2015 at 7:01 PM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

no I meant the source is for the company I work for, not my personal
project.

On Tue, Jun 9, 2015 at 6:58 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

On Tue, Jun 9, 2015 at 12:43 PM, Arbi Akhina <arbi.akhina(a)gmail.com>
wrote:

I don't own the source, but here is the jar file
http://tempsend.com/6D745B7B07/7033/modules.jar
you can try to run it locally with java -jar modules.jar then
connect to localhost:8080
Hmm, OK. I thought you had a minimal example from you last note.
I'm not really up for running untrusted binary code. If you can put a
minimal code sample together, I'll give it a try.



An here is the manifest.yml i'm using:
---
applications:
- name: modules
memory: 4G
disk_quota: 2G
timeout: 180
instances: 1
host: modules-${random-word}
path: modules.jar
buildpack: https://github.com/cloudfoundry/java-buildpack
command: sleep 2 &&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY com.heavenize.osmoze.kernel.HelloHandler
# env:
# JAVA_OPTS: "$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000"

I'm having both files in the same folder modules/ and I push with cf
push
That seems OK. Is this app designed to run on CF? By that, I mean
is it configure to log to STDOUT / STDERR and is it going to listen on the
right port (i.e $PORT)?

Dan




Thanks a lot

On Tue, Jun 9, 2015 at 6:25 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

I'd be happy to try the demo and see if it works for me. Can you
post a complete project on github so I can just `git clone` & build?

Dan

On Tue, Jun 9, 2015 at 12:21 PM, Arbi Akhina <arbi.akhina(a)gmail.com
wrote:
I've tried with a simple app that just launches an embedded jetty
that listen on the vcap port and it worked (I can connect to the app url
from curl and chrome)
then I moved this single file into my modules.jar (the real app)
and configure it as main class, but here it doesn't work.

here is the class in question:
public class HelloHandler extends AbstractHandler
{
public static void main(String[] args)
{
int VCAP_APP_PORT =
Integer.parseInt((System.getenv("VCAP_APP_PORT") != null ?
System.getenv("VCAP_APP_PORT") : "8080"));

Server server = new Server(VCAP_APP_PORT);
server.setHandler(new HelloHandler());

try
{
server.start();
server.join();
}
catch (Exception exception)
{
System.out.println("Failed to start embedded jetty server.");
exception.printStackTrace();
}
}

final String greeting;
final String body;

public HelloHandler()
{
this("Hello World");
}

public HelloHandler( String greeting )
{
this(greeting, null);
}

public HelloHandler( String greeting, String body )
{
this.greeting = greeting;
this.body = body;
}

public void handle( String target,
Request baseRequest,
HttpServletRequest request,
HttpServletResponse response ) throws
IOException,

ServletException
{
response.setContentType("text/html; charset=utf-8");
response.setStatus(HttpServletResponse.SC_OK);

PrintWriter out = response.getWriter();

out.println("<h1>" + greeting + "</h1>");
out.println("<p>Current directory: " + getCurrentDirectory()
+ "</p>");
out.println("<p> Parent directory: " + getParentDirectory()
+ "</p>");
for(File file: getFiles()) {
out.println("<p>" + file.getAbsolutePath() + "</p>");
}

if (body != null)
{
out.println(body);
}

baseRequest.setHandled(true);
}
private List<File> getFiles() {
return Arrays.asList(new File(".").listFiles());
}

private String getCurrentDirectory()
{
String directory = null;
try
{
directory = new File(".").getCanonicalPath();
}
catch (Exception e)
{
e.printStackTrace();
}
return directory;
}

private String getParentDirectory()
{
String directory = null;
try
{
directory = new File("..").getCanonicalPath();
}
catch (Exception e)
{
e.printStackTrace();
}
return directory;
}
}


On Tue, Jun 9, 2015 at 5:34 PM, Daniel Mikusa <dmikusa(a)pivotal.io>
wrote:

Double check that your app is configured to log to STDOUT /
STDER. If it's logging to a file, you won't see the messages since the
system only captures STDOUT & STDERR.

Also, you might try a demo app like spring-music, just to make
sure your install is working OK.

https://github.com/cloudfoundry-samples/spring-music

Dan

On Tue, Jun 9, 2015 at 10:40 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Same logs I still can't see what went wrong!

On Tue, Jun 9, 2015 at 3:58 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

And so you'd want to set `command` to `sleep 2 &&
CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/
bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m.. -memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java
-cp $PWD/. -Djava.io.tmpdir=$TMPDIR -XX:OnOutOfMemoryError=$PWD/.
java-buildpack/open_jdk_jre/bin/killjava.sh $CALCULATED_MEMORY
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000
com.heavenize.osmoze.kernel.HelloHandler`.

Dan

On Tue, Jun 9, 2015 at 9:29 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Here is the content of *detected_start_command*:
$ CF_TRACE=true cf app modules | grep "detected_start_command"
"detected_start_command":
"CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler",

{"guid":"2dabf3c6-1736-4eb4-9bb7-40dc58bce246","name":"modules","routes":[{"guid":"014f77ed-e450-4209-be23-40e4a8257cfc","host":"modules-rhythmic-uvulatomy","domain":{"guid":"a77afa99-76d6-4f3f-9e07-27775597709f","name":"
10.244.0.34.xip.io
"}}],"running_instances":0,"services":[],"available_domains":[{"guid":"a77afa99-76d6-4f3f-9e07-27775597709f","name":"
10.244.0.34.xip.io
"}],"name":"modules","production":false,"space_guid":"381707f7-88d4-4f6e-bd7c-80d7f0699b0f","stack_guid":"3431865a-f165-4e75-9221-4f418e9de889","buildpack":"
https://github.com/cloudfoundry/java-buildpack","detected_buildpack":null,"environment_json":{"JAVA_OPTS":"$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=192.168.2.8:8000"},"memory":4096,"instances":1,"disk_quota":2048,"state":"STARTED","version":"e8f0e18e-7c09-4143-a17c-83a6e32eed84","command":null,"console":false,"debug":null,"staging_task_id":"1249276465c64c1884d401452d2365af","package_state":"STAGED","health_check_type":"port","health_check_timeout":180,"staging_failed_reason":null,"diego":false,"docker_image":null,"package_updated_at":"2015-06-08T07:56:44Z","detected_start_command":"CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-1.1.1_RELEASE
-memorySizes=metaspace:64m..
-memoryWeights=heap:75,metaspace:10,stack:5,native:10
-totMemory=$MEMORY_LIMIT) && $PWD/.java-buildpack/open_jdk_jre/bin/java -cp
$PWD/. -Djava.io.tmpdir=$TMPDIR
-XX:OnOutOfMemoryError=$PWD/.java-buildpack/open_jdk_jre/bin/killjava.sh
$CALCULATED_MEMORY -agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000 com.heavenize.osmoze.kernel.HelloHandler"}


On Tue, Jun 9, 2015 at 1:35 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

On Tue, Jun 9, 2015 at 5:05 AM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

Hi, thanks for the hint, I tried to add to the manifest the
following entry:
command: sleep 2 && java -jar modules.jar
The Java build pack does not put Java on the path, which is
what you're seeing in the error below. I will usually run `CF_TRACE=true
cf app <app-name> | grep "detected_start_command"` which shows the command
the build pack detected and then just copy & paste that into my custom
command.



but it looks like it's not appropriate as I see in the logs:
2015-06-08T05:34:03.71+0200 [App/0] ERR bash: java:
command not found

On Mon, Jun 8, 2015 at 6:32 PM, Daniel Mikusa <
dmikusa(a)pivotal.io> wrote:

On Mon, Jun 8, 2015 at 12:03 PM, Arbi Akhina <
arbi.akhina(a)gmail.com> wrote:

I'm trying to push an executable JAR to a bosh-lite
instance, on the logs I see CF trying many times to restart the app and
eventually fail.

I can't find out why the app crashes as there is no app
logs returned by CF. I tried to remotely debug the app (as described in
[1]) but nothing happens on eclipse. Any hint to solve this issue is
appreciated.
There's a known issue that occurs when an app starts and
fails in rapid succession and results in the log entries being missed. If
you add a couple second pause into the app, it will give the system enough
time to attach the logger and you'll see the output generated by the
crashing app.

You can do this with a custom start command `cf push -c
'sleep 2 && <normal-cmd>'` or with a `.profile.d` script that sleeps for a
couple seconds.

Dan




1. I'm launching the app with:
*cf push -t 180*

2. Here is the manifest.yml content:
---
applications:
- name: modules
memory: 4G
instances: 1
host: modules-${random-word}
path: modules.jar
buildpack:
https://github.com/cloudfoundry/java-buildpack
env:
JAVA_OPTS: "$JAVA_OPTS
-agentlib:jdwp=transport=dt_socket,address=
192.168.2.8:8000"

3. Here is the log:
Connecté, le dumping journaux récents pour application
modules en org heavenize / espace dev comme admin...

2015-06-07T12:05:15.69+0200 [API] OUT Created app with
guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:05:16.10+0200 [API] OUT Updated app with
guid 1fbf3378-6512-46de-bae4-02ee30275464
({"route"=>"68e27d8d-4ff6-443b-a3e0-416c40d325d3"})
2015-06-07T12:12:21.55+0200 [DEA] OUT Got staging
request for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:12:25.40+0200 [API] OUT Updated app with
guid 1fbf3378-6512-46de-bae4-02ee30275464 ({"state"=>"STARTED"})
2015-06-07T12:12:38.08+0200 [STG] OUT ----->
Downloaded app package (163M)
2015-06-07T12:13:40.93+0200 [STG] ERR Cloning into
'/tmp/buildpacks/java-buildpack'...
2015-06-07T12:14:01.67+0200 [STG] OUT -----> Java
Buildpack Version: c862ac8 |
https://github.com/cloudfoundry/java-buildpack#c862ac8
2015-06-07T12:14:13.89+0200 [STG] OUT ----->
Downloading Open Jdk JRE 1.8.0_45 from
https://download.run.pivotal.io/openjdk/lucid/x86_64/openjdk-1.8.0_45.tar.gz
(11.6s)
2015-06-07T12:14:15.34+0200 [STG] OUT Expanding
Open Jdk JRE to .java-buildpack/open_jdk_jre (1.4s)
2015-06-07T12:14:15.89+0200 [STG] OUT ----->
Downloading Open JDK Like Memory Calculator 1.1.1_RELEASE from
https://download.run.pivotal.io/memory-calculator/lucid/x86_64/memory-calculator-1.1.1_RELEASE
(0.5s)
2015-06-07T12:14:15.90+0200 [STG] OUT Memory
Settings: -XX:MaxMetaspaceSize=419430K -XX:MetaspaceSize=419430K -Xss1M
-Xmx3G -Xms3G
2015-06-07T12:15:43.44+0200 [STG] OUT -----> Uploading
droplet (151M)
2015-06-07T12:16:07.51+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:16:29.66+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"9d55c5f791324d358bffb4c961a4c7ee", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672189}
2015-06-07T12:17:14.18+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:31.10+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"2ae0c26f33864f40989ee870a1b9e3db", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672251}
2015-06-07T12:17:38.48+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:38.48+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:38.48+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:17:55.31+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:11.82+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"dc872d38f3324af481c82ba67f0e216c", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672291}
2015-06-07T12:18:18.69+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:18.69+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:18.69+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:34.06+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:50.98+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"623c3af7e3e84801b6fd44eeee9c0a12", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672330}
2015-06-07T12:18:58.80+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:58.80+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:18:58.80+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:34.08+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:50.36+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"40727eea293146948af197e13443843c", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672390}
2015-06-07T12:19:59.01+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:59.01+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:19:59.01+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:04.12+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:20.61+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"f7ffff55692a418c847f4f37be574ddf", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672480}
2015-06-07T12:21:29.43+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:29.43+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:21:29.47+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:34.16+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:49.97+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"4581e97c6b0f4504b8d64a5c69d6787b", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672629}
2015-06-07T12:23:50.29+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:50.29+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:23:50.29+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:14.24+0200 [DEA] OUT Starting app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:29.82+0200 [API] OUT App instance
exited with guid 1fbf3378-6512-46de-bae4-02ee30275464 payload:
{"cc_partition"=>"default",
"droplet"=>"1fbf3378-6512-46de-bae4-02ee30275464",
"version"=>"703f793c-e388-41ac-b1f1-c564b301ca70",
"instance"=>"f98749490a6743598f57d3848eb06177", "index"=>0,
"reason"=>"CRASHED", "exit_status"=>-1, "exit_description"=>"failed to
start", "crash_timestamp"=>1433672909}
2015-06-07T12:28:31.73+0200 [DEA] OUT Removing crash
for app with id 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:31.73+0200 [DEA] OUT Stopping app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464
2015-06-07T12:28:31.73+0200 [DEA] OUT Stopped app
instance (index 0) with guid 1fbf3378-6512-46de-bae4-02ee30275464



[1]
http://docs.cloudfoundry.org/buildpacks/java/java-tips.html#debugging



_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


Re: How random is Metron's Doppler selection?

Mike Youngstrom <youngm@...>
 

Or better yet support a syslog endpoint in an app container that sends to
loggregator. Then we can get full stack traces in a single event. :)

Mike

On Thu, Jun 11, 2015 at 4:55 PM, Mike Heath <elcapo(a)gmail.com> wrote:

Actually, this might explain why some of our customers are so frustrated
trying to read their stack traces in Splunk. :\

So each line of a stack trace could go to a different Doppler. That means
each line of the stack trace goes out to a different syslog drain making it
impossible to consolidate that stack trace into a single logging event when
passed off to a third-party logging system like Splunk. This sucks. To be
fair, Splunk has never been any good at dealing with stack traces.

What are the possibilities of getting some kind of optionally enabled
application instance affinity put into Metron? (I know. I know. I can
submit a PR.)

-Mike

On Thu, Jun 11, 2015 at 3:54 PM, John Tuley <jtuley(a)pivotal.io> wrote:

Oops, wrong link. Should be
https://github.com/cloudfoundry/loggregator/blob/develop/src/metron/main.go#L188-L197
.

Sorry about that!

– John Tuley

On Thu, Jun 11, 2015 at 3:36 PM, John Tuley <jtuley(a)pivotal.io> wrote:

Mike,

Metron chooses a randomly-available Doppler for each message
<https://www.pivotaltracker.com/story/show/96801752>. Availability
prefers same-zone Doppler servers:

- If a Metron instance knows about any same-zone Dopplers, it
chooses one at random for each message.
- If no same-zone Dopplers are present, the random choice is made
from the list of all known servers.


In fact, the behavior you describe is the behavior of DEA Logging Agent
before Metron existed. What we discovered with that approach is that it
balances load very unfairly, as a single high-volume app is stuck on one
server. While the "new" mechanism does not guarantee consistency, it does
enable the Doppler pool to more-evenly share load.

If you're seeing that a single app instance is routed to the same
Doppler server every time, then (without further information) I would guess
that you're either running a single Doppler instance in each availability
zone, or your deck is stacked. :-) If neither of those is true and you're
still observing that Metron routes messages from an app instance to a
single Doppler, I'd love to investigate how that is happening.

– John Tuley

On Thu, Jun 11, 2015 at 2:45 PM, Mike Heath <elcapo(a)gmail.com> wrote:

Metron's documentation [1] says "All Metron traffic is randomly
distributed across available Dopplers." How random is this? Based on
observation, it appears that logs for an individual application instance
are consistently sent to the same Doppler instance. The consistency aspect
is very important for us so that our Syslog forwarder can consolidate stack
traces into a single logging event.

How random is this distribution really for an application instance's
logs?

-Mike

1 - https://github.com/cloudfoundry/loggregator/tree/develop/src/metron

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev