Hi Jim,

So, to be clear what we're basically doing is using unicode newline
character to fool loggregator (which is looking for \n) into thinking that
it isn't a new log event right? Does \u2028 work as a new line character
when tailing logs in the CLI? Anyone tried this unicode new line character
in various consoles? IDE, xterm, etc? I'm wondering if developers will
need to have different config for development.

Mike

On Mon, Mar 14, 2016 at 12:17 PM, Jim CF Campbell <jcampbell(a)pivotal.io>
wrote:

Hi Mike and Alex,

Two things - for Java, we are working toward defining an enhanced metric
format that will support transport of Multi Lines.

The second is this workaround that David Laing suggested for Logstash.
Think you could use it for Splunk?

With the Java Logback library you can do this by adding
"%replace(%xException){'\n','\u2028'}%nopex" to your logging config[1] ,
and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana will
display as a new line.

mutate {

gsub => [ "[@message]", '\u2028', "

"]
^^^ Seems that passing a string with an actual newline in it is the only
way to make gsub work

}

to replace the token with a regular newline again so it displays
"properly" in Kibana.

[1] github.com/dpin...ication.yml#L12
<https://github.com/dpinto-pivotal/cf-SpringBootTrader-config/blob/master/application.yml#L12>

[2] github.com/logs...se.conf#L60-L64
<https://github.com/logsearch/logsearch-for-cloudfoundry/blob/master/src/logsearch-config/src/logstash-filters/snippets/firehose.conf#L60-L64>


On Mon, Mar 14, 2016 at 11:11 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:

I'll let the Loggregator team respond formally. But, in my
conversations with the Loggregator team I think we're basically stuck not
sure what the right thing to do is on the client side. How does the client
trigger in loggregator that this is a multi line log message or what is the
right way for loggregator to detect that the client is trying to send a
multi line log message? Any ideas?

Mike

On Mon, Mar 14, 2016 at 10:25 AM, Aliaksandr Prysmakou <
prysmakou(a)gmail.com> wrote:

Hi guys,
Are there any updates about "Multiline Log Entry" issue? How correctly
deal with stacktraces?
Links to the tracker to read?
----
Alex Prysmakou / Altoros
Tel: (617) 841-2121 ext. 5161 | Toll free: 855-ALTOROS
Skype: aliaksandr.prysmakou
www.altoros.com | blog.altoros.com | twitter.com/altoros

--
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io |
303.618.0963

This feature allows developers more control of the staging and deployment
of their application code, without them having to fork existing buildpacks
or create their own.


https://docs.google.com/document/d/1PnTtTLwXOTG7f70ilWGlbTbi1LAXZu9zYnrUVvjr31I/edit

Hooks give developers the ability to optionally:
* run scripts in the staging container before and/or after the
bin/compile scripts executed by the buildpack, and
* run scripts in each app container before the app starts (via .profile
as per the Heroku buildpack API)

A similar feature has been available and used extensively in Stackato for
a few years, and we'd like to contribute this functionality back to Cloud
Foundry.

A proof-of-concept of this feature has already been submitted as a pull
request, and the Feature Narrative addresses many of the questions raised
in the PR discussion:

https://github.com/cloudfoundry-incubator/buildpack_app_lifecycle/pull/13

Please weigh in with comments in the document itself or in this thread.

Thanks,

TT

Hi there,

The CF Buildpacks team is considering taking on a line of work to provide
more formal support for multibuildpacks. Before we start, we would be
interested in learning if any community users have compelling use cases
they could share with us.

For more information on multibuildpacks, see Heroku's documentation [1]

[1] -
https://devcenter.heroku.com/articles/using-multiple-buildpacks-for-an-app

Le 11 avr. 2016 à 18:15, Benjamin Gandon <benjamin(a)gandon.org> a écrit :

Hi cf devs,


I’m running a CF deployment with redundancy, and I just experienced my consul servers not being able to elect any leader.
That’s a VERY frustrating situation that keeps the whole CF deployment down, until you get a deeper understanding of consul, and figure out they just need a silly manual 'join' so that they get back together.

But that was definitely not easy to nail down because at first look, I could just see monit restarting the “agent_ctl” every 60 seconds because confab was not writing the damn PID file.


More specifically, the 3 consul servers (i.e. consul_z1/0, consul_z1/1 and consul_z2/0) had properly left oneanother uppon a graceful shutdown. This state was persisted in /var/vcap/store/raft/peers.json being “null” on each one of them, so they would not get back together on restart. A manual 'join' was necessary. But it took me hours to get there because I’m no expert with consul.

And until the 'join' is made, VerifySynced() was negative in confab, and monit was constantly starting and stopping it every 60 seconds. But once you step back, you realize confab was actually waiting for the new leader to be elected before it writes the PID file. Which is questionable.

So, I’m asking 3 questions here:

1. Does writing the PID file in confab that late really makes sense?
2. Could someone please write some minimal documentation about confab, at least to tell what it is supposed to do?
3. Wouldn’t it be wiser that whenever any of the consul servers is not here, then the cluster gets unhealthy?

With this 3rd question, I mean that even on a graceful TERM or INT, no consul server should not perform any graceful 'leave'. With this different approach, then they would properly be back up even when performing a complete graceful restart of the cluster.

This can be done with those extra configs from the “confab” wrapper:

{
"skip_leave_on_interrupt": true,
"leave_on_terminate": false
}

What do you guys think of it?


/Benjamin

Le 11 avr. 2016 à 17:46, Eric Malm <emalm(a)pivotal.io> a écrit :

Hi, Benjamin,

Yes, the BOSH-Lite boxes with kernel 3.19.0-40 through 3.19.0-50 are all susceptible to the AUFS bug. Kernel versions 3.19.0-51 and later will be fine, and I believe the earliest BOSH-Lite Vagrant box with one of those kernel versions is 9000.102.0. The 3.19.0-49 kernel that went into 3192 was a one-off build that Canonical supplied in advance of the release of the official kernel package with the fix (https://launchpad.net/ubuntu/+source/linux-lts-vivid/3.19.0-51.57~14.04.1 <https://launchpad.net/ubuntu/+source/linux-lts-vivid/3.19.0-51.57~14.04.1>), and the 'official' package with kernel 3.19.0-49 still has the AUFS bug.

Thanks,
Eric

On Mon, Apr 11, 2016 at 8:36 AM, Benjamin Gandon <benjamin(a)gandon.org <mailto:benjamin(a)gandon.org>> wrote:
Hi,

Sorry for the late up, but would this hit bosh-lite too?
Because after it has run for a while, I’m experiencing severe similar issues with the 53 garden containers I use in Bosh-Lite.

Config :
- Bosh-lite v9000.91.0 (i.e. bosh v250 + warden-cpi v29 + garden-linux v0.331.0) and the kernel is 3.19.0-47.53~14.04.1 (I might have upgraded it)
- Deployment: cf v231 + Diego v0.1434.0 + Garden-linux v0.333.0 + Etcd v36 + cf-mysql v26 + other

Will the linux-image-3.19.0-49-generic fix the issue, as it was done in this 2016-02-08 commit <https://github.com/cloudfoundry/bosh/commit/750c5e7ed70b1d7753500ca725590c1c0eac1262> for stemcell 3192 ?

As a safety measure, I decided to upgrade to kernel 3.19.0-58-generic and I would be happy to get a confirmation that (1) my bosh-lite deployment was hit by the AUFS bug, and that (2) the new kernel I installed will get me off this operational nightmare.

Thanks!

Le 28 janv. 2016 à 02:06, Eric Malm <emalm(a)pivotal.io <mailto:emalm(a)pivotal.io>> a écrit :

Hi, Mike,

Warden also uses aufs for its containers' overlay filesystems, so we expect the same issue to affect the DEAs on these stemcell versions. I'm not aware of a deliberate attempt to reproduce it on the DEAs, though.

Thanks,
Eric

On Wed, Jan 27, 2016 at 4:08 PM, Mike Youngstrom <youngm(a)gmail.com <mailto:youngm(a)gmail.com>> wrote:
Thanks Will. Does anyone know if this bug could also impacts Warden?

Mike

On Wed, Jan 27, 2016 at 9:50 AM, Will Pragnell <wpragnell(a)pivotal.io <mailto:wpragnell(a)pivotal.io>> wrote:
A bug with AUFS [1] was introduced in version 3.19.0-40 of the linux kernel. This bug can cause containers to end up with unkillable zombie processes with high CPU usage. This can happen any time a container is supposed to be destroyed.

This affects both Garden-Linux and Warden (and Docker). If you see significant slowdown or increased CPU usage on DEAs or Diego cells, it might well be this. It will probably build slowly up over time, so you may not notice anything for a while depending on the rate of app instance churn on your deployment.

The bad version of the kernel is present in stemcell 3160 and later. I can't recommend using older stemcells because the bad kernel versions also include fixes for several high severity security vulnerabilities (at least [2-5], there may be others I've missed). Were it not for these, rolling back to stemcell 3157 would be the fix.

We're waiting for a fix to make its way into the kernel, and the BOSH team will produce a stemcell with the fix as soon as possible. In the meantime, I'd suggest simply keeping a closer eye than usual on your DEAs and Diego cells.

If this issue occurs, the only solution is to recreate that machine. While we've not had any actual reports of this issue occurring for Cloud Foundry deployments in the wild yet, we're confident that the issue will be occurring. The Diego team have seen it in testing, and several teams have encountered the issue with their Concourse workers, which also use Garden-Linux.

As always, please get in touch out if you have any questions.

Will - Garden PM

[1]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1533043 <https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1533043>
[2]: http://www.ubuntu.com/usn/usn-2857-1/ <http://www.ubuntu.com/usn/usn-2857-1/>
[3]: http://www.ubuntu.com/usn/usn-2868-1/ <http://www.ubuntu.com/usn/usn-2868-1/>
[4]: http://www.ubuntu.com/usn/usn-2869-1/ <http://www.ubuntu.com/usn/usn-2869-1/>
[5]: http://www.ubuntu.com/usn/usn-2871-2/ <http://www.ubuntu.com/usn/usn-2871-2/>

Le 28 janv. 2016 à 02:06, Eric Malm <emalm(a)pivotal.io> a écrit :

Hi, Mike,

Warden also uses aufs for its containers' overlay filesystems, so we expect the same issue to affect the DEAs on these stemcell versions. I'm not aware of a deliberate attempt to reproduce it on the DEAs, though.

Thanks,
Eric

On Wed, Jan 27, 2016 at 4:08 PM, Mike Youngstrom <youngm(a)gmail.com <mailto:youngm(a)gmail.com>> wrote:
Thanks Will. Does anyone know if this bug could also impacts Warden?

Mike

On Wed, Jan 27, 2016 at 9:50 AM, Will Pragnell <wpragnell(a)pivotal.io <mailto:wpragnell(a)pivotal.io>> wrote:
A bug with AUFS [1] was introduced in version 3.19.0-40 of the linux kernel. This bug can cause containers to end up with unkillable zombie processes with high CPU usage. This can happen any time a container is supposed to be destroyed.

This affects both Garden-Linux and Warden (and Docker). If you see significant slowdown or increased CPU usage on DEAs or Diego cells, it might well be this. It will probably build slowly up over time, so you may not notice anything for a while depending on the rate of app instance churn on your deployment.

The bad version of the kernel is present in stemcell 3160 and later. I can't recommend using older stemcells because the bad kernel versions also include fixes for several high severity security vulnerabilities (at least [2-5], there may be others I've missed). Were it not for these, rolling back to stemcell 3157 would be the fix.

We're waiting for a fix to make its way into the kernel, and the BOSH team will produce a stemcell with the fix as soon as possible. In the meantime, I'd suggest simply keeping a closer eye than usual on your DEAs and Diego cells.

If this issue occurs, the only solution is to recreate that machine. While we've not had any actual reports of this issue occurring for Cloud Foundry deployments in the wild yet, we're confident that the issue will be occurring. The Diego team have seen it in testing, and several teams have encountered the issue with their Concourse workers, which also use Garden-Linux.

As always, please get in touch out if you have any questions.

Will - Garden PM

[1]: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1533043 <https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1533043>
[2]: http://www.ubuntu.com/usn/usn-2857-1/ <http://www.ubuntu.com/usn/usn-2857-1/>
[3]: http://www.ubuntu.com/usn/usn-2868-1/ <http://www.ubuntu.com/usn/usn-2868-1/>
[4]: http://www.ubuntu.com/usn/usn-2869-1/ <http://www.ubuntu.com/usn/usn-2869-1/>
[5]: http://www.ubuntu.com/usn/usn-2871-2/ <http://www.ubuntu.com/usn/usn-2871-2/>

Thanks, Tom. The errors about the 401 response code make me suspect that
the nsync-bulker doesn't have the correct basic-auth credentials for the
internal app-enumeration endpoint it queries on CC. Could you check whether
the diego.nsync.cc.basic_auth_username
and diego.nsync.cc.basic_auth_password properties in your Diego manifest
are the same as the cc.internal_api_user and cc.internal_api_password
properties in your CF manifest? There was also a previous pair of CF/Diego
release versions where those properties had different defaults for the user
names in the job specs, but I believe they match in CF v230 and Diego
v0.1450.0.

Best,
Eric

On Fri, Apr 8, 2016 at 5:33 PM, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:

Yes, the logs are there.
I grepped the logs for error. I see a lot of:


{"timestamp":"1460161846.254659176","source":"nsync-bulker","message":"nsync-bulker.sync.not-bumping-freshness-because-of","log_level":2,"data":{"
error":"invalid response code 401","session":"6713"}}


{"timestamp":"1460161876.286883593","source":"nsync-bulker","message":"nsync-bulker.sync.not-bumping-freshness-because-of","log_level":2,"data":{"
error":"invalid response code 401","session":"6714"}}


{"timestamp":"1460161906.315121412","source":"nsync-bulker","message":"nsync-bulker.sync.not-bumping-freshness-because-of","log_level":2,"data":{"
error":"invalid response code 401","session":"6715"}}


{"timestamp":"1460161936.352133274","source":"nsync-bulker","message":"nsync-bulker.sync.not-bumping-freshness-because-of","log_level":2,"data":{"
error":"invalid response code 401","session":"6716"}}


{"timestamp":"1460161966.383990765","source":"nsync-bulker","message":"nsync-bulker.sync.not-bumping-freshness-because-of","log_level":2,"data":{"
error":"invalid response code 401","session":"6717"}}


Let me know if there's something specific you wish to find.


Tom

On Fri, Apr 8, 2016 at 11:47 AM, Eric Malm <emalm(a)pivotal.io> wrote:

Thanks, Tom, glad you were able to use veritas to find and remove the
stray apps. I'd like to know how they remained present in the first place.
Do you have logs from the nsync-bulker jobs on the cc_bridge VMs in your
deployment? That BOSH job has the responsibility of updating the Diego
DesiredLRPs to match the current set of CF apps, so if there are
synchronization errors they should be present in those logs.

Thanks,
Eric, CF Runtime Diego PM

On Fri, Apr 8, 2016 at 8:09 AM, Kris Hicks <khicks(a)pivotal.io> wrote:

It would be nice to figure out the root cause here.

Does having two crashed and two apps have some significance as to why
the delete failed, though appeared successful?

On Friday, April 8, 2016, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:

Thank you.

Veritas is quite informative. I found 2 apps running and 2 crashed.
I deleted them and all appears well.


On Mon, Apr 4, 2016 at 7:03 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Ok, I would use veritas to look at the Diego BBS, and confirm that it
still thinks the app is there. You can also go onto the router and query
its HTTP endpoint to confirm that the route you're seeing is also still
there: https://github.com/cloudfoundry/gorouter#instrumentation.
Lastly I would connect to the CCDB and confirm that the app and route are
*not* there. This will reduce the problem to figuring out why Diego isn't
being updated to know that the non-existing app is no longer desired.

On Mon, Apr 4, 2016 at 3:47 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

The route still exists. I was reluctant to delete it and have the
"app" still running. I wanted some way to track it down, not that it has
helped, other than let me know it is still running.

Pushed the app, with a different name/host, with no problems and it
runs as it should.

On Mon, Apr 4, 2016 at 6:17 PM, Amit Gupta <agupta(a)pivotal.io>
wrote:

Tom,

So you're saying that none of the org/spaces shows the app or the
route, but the app continues to run and be routeable?

I could imagine this happen if some CC Bridge components are not
able to talk to either CC or Diego BBS, leaving the data in the Diego BBS
stale. In the case of stale info, Diego may not know that the LRP is no
longer desired, and it will do the safe thing of keeping it around, and
emitting its route to the gorouter, which just does what it's told (it
doesn't check whether CC knows about the route or not).

Are you able to push new apps or delete other apps with the Diego
backend?

Amit

On Fri, Apr 1, 2016 at 1:00 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

JT,

Thanks for responding.

This is a test runtime and small. I checked all orgs and spaces.
No routes matching the app.

Found the route information and the result:

{

"total_results": 0,

"total_pages": 1,

"prev_url": null,

"next_url": null,

"resources": []

}

To learn what the output may look like, I check existing routes
with apps and without. The output appears to be the same as if the app has
been deleted.

Even now, the app url still returns a page from the app, even
though it is deleted.

Thanks,

Tom

On Fri, Apr 1, 2016 at 1:52 PM, JT Archie <jarchie(a)pivotal.io>
wrote:

Tom,

Are you sure the route isn't bound to another application in
another org/space?

When you do `cf routes` it only show routes for the current
space. You can hit specific API endpoints though to get all the apps for a
route.

For example, `cf
curl /v2/routes/89fc2a5e-3a9b-4a88-a360-e405cdbd6f87/apps` will show all
the apps for a particular route. Obviously replacing the route ID with the
correct ID. To find that, I recommend going through `CF_TRACE=true cf
routes` and grabbing the ID.

Let see if you can hunt it down that way.

Kind Regards,

JT

On Fri, Apr 1, 2016 at 8:51 AM, Tom Sherrod <
tom.sherrod(a)gmail.com> wrote:

cf 230, diego 0.1450.0, etcd 27, garden-linux 0.330.0
Default to diego true.

Developer deployed a java application. Deleted the application:
cf delete <app> No errors.
The app still responds. The only thing left is the route.
I've not encountered this before. Delete has been delete and
even if route remains, 404 Not Found: Requested route ('<hostname.domain>')
does not exist. is returned.

Pointers on tracking this down appreciated.

Tom

Yes, the logs are there.
I grepped the logs for error. I see a lot of:


{"timestamp":"1460161846.254659176","source":"nsync-bulker","message":"nsync-bulker.sync.not-bumping-freshness-because-of","log_level":2,"data":{"
error":"invalid response code 401","session":"6713"}}


{"timestamp":"1460161876.286883593","source":"nsync-bulker","message":"nsync-bulker.sync.not-bumping-freshness-because-of","log_level":2,"data":{"
error":"invalid response code 401","session":"6714"}}


{"timestamp":"1460161906.315121412","source":"nsync-bulker","message":"nsync-bulker.sync.not-bumping-freshness-because-of","log_level":2,"data":{"
error":"invalid response code 401","session":"6715"}}


{"timestamp":"1460161936.352133274","source":"nsync-bulker","message":"nsync-bulker.sync.not-bumping-freshness-because-of","log_level":2,"data":{"
error":"invalid response code 401","session":"6716"}}


{"timestamp":"1460161966.383990765","source":"nsync-bulker","message":"nsync-bulker.sync.not-bumping-freshness-because-of","log_level":2,"data":{"
error":"invalid response code 401","session":"6717"}}


Let me know if there's something specific you wish to find.


Tom

On Fri, Apr 8, 2016 at 11:47 AM, Eric Malm <emalm(a)pivotal.io> wrote:

Thanks, Tom, glad you were able to use veritas to find and remove the
stray apps. I'd like to know how they remained present in the first place.
Do you have logs from the nsync-bulker jobs on the cc_bridge VMs in your
deployment? That BOSH job has the responsibility of updating the Diego
DesiredLRPs to match the current set of CF apps, so if there are
synchronization errors they should be present in those logs.

Thanks,
Eric, CF Runtime Diego PM

On Fri, Apr 8, 2016 at 8:09 AM, Kris Hicks <khicks(a)pivotal.io> wrote:

It would be nice to figure out the root cause here.

Does having two crashed and two apps have some significance as to why
the delete failed, though appeared successful?

On Friday, April 8, 2016, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:

Thank you.

Veritas is quite informative. I found 2 apps running and 2 crashed.
I deleted them and all appears well.


On Mon, Apr 4, 2016 at 7:03 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Ok, I would use veritas to look at the Diego BBS, and confirm that it
still thinks the app is there. You can also go onto the router and query
its HTTP endpoint to confirm that the route you're seeing is also still
there: https://github.com/cloudfoundry/gorouter#instrumentation.
Lastly I would connect to the CCDB and confirm that the app and route are
*not* there. This will reduce the problem to figuring out why Diego isn't
being updated to know that the non-existing app is no longer desired.

On Mon, Apr 4, 2016 at 3:47 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

The route still exists. I was reluctant to delete it and have the
"app" still running. I wanted some way to track it down, not that it has
helped, other than let me know it is still running.

Pushed the app, with a different name/host, with no problems and it
runs as it should.

On Mon, Apr 4, 2016 at 6:17 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Tom,

So you're saying that none of the org/spaces shows the app or the
route, but the app continues to run and be routeable?

I could imagine this happen if some CC Bridge components are not
able to talk to either CC or Diego BBS, leaving the data in the Diego BBS
stale. In the case of stale info, Diego may not know that the LRP is no
longer desired, and it will do the safe thing of keeping it around, and
emitting its route to the gorouter, which just does what it's told (it
doesn't check whether CC knows about the route or not).

Are you able to push new apps or delete other apps with the Diego
backend?

Amit

On Fri, Apr 1, 2016 at 1:00 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

JT,

Thanks for responding.

This is a test runtime and small. I checked all orgs and spaces. No
routes matching the app.

Found the route information and the result:

{

"total_results": 0,

"total_pages": 1,

"prev_url": null,

"next_url": null,

"resources": []

}

To learn what the output may look like, I check existing routes
with apps and without. The output appears to be the same as if the app has
been deleted.

Even now, the app url still returns a page from the app, even
though it is deleted.

Thanks,

Tom

On Fri, Apr 1, 2016 at 1:52 PM, JT Archie <jarchie(a)pivotal.io>
wrote:

Tom,

Are you sure the route isn't bound to another application in
another org/space?

When you do `cf routes` it only show routes for the current space.
You can hit specific API endpoints though to get all the apps for a route.

For example, `cf
curl /v2/routes/89fc2a5e-3a9b-4a88-a360-e405cdbd6f87/apps` will show all
the apps for a particular route. Obviously replacing the route ID with the
correct ID. To find that, I recommend going through `CF_TRACE=true cf
routes` and grabbing the ID.

Let see if you can hunt it down that way.

Kind Regards,

JT

On Fri, Apr 1, 2016 at 8:51 AM, Tom Sherrod <tom.sherrod(a)gmail.com

wrote:

cf 230, diego 0.1450.0, etcd 27, garden-linux 0.330.0
Default to diego true.

Developer deployed a java application. Deleted the application:
cf delete <app> No errors.
The app still responds. The only thing left is the route.
I've not encountered this before. Delete has been delete and even
if route remains, 404 Not Found: Requested route ('<hostname.domain>') does
not exist. is returned.

Pointers on tracking this down appreciated.

Tom

Thanks, Tom, glad you were able to use veritas to find and remove the
stray apps. I'd like to know how they remained present in the first place.
Do you have logs from the nsync-bulker jobs on the cc_bridge VMs in your
deployment? That BOSH job has the responsibility of updating the Diego
DesiredLRPs to match the current set of CF apps, so if there are
synchronization errors they should be present in those logs.

Thanks,
Eric, CF Runtime Diego PM

On Fri, Apr 8, 2016 at 8:09 AM, Kris Hicks <khicks(a)pivotal.io> wrote:

It would be nice to figure out the root cause here.

Does having two crashed and two apps have some significance as to why the
delete failed, though appeared successful?

On Friday, April 8, 2016, Tom Sherrod <tom.sherrod(a)gmail.com> wrote:

Thank you.

Veritas is quite informative. I found 2 apps running and 2 crashed.
I deleted them and all appears well.


On Mon, Apr 4, 2016 at 7:03 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Ok, I would use veritas to look at the Diego BBS, and confirm that it
still thinks the app is there. You can also go onto the router and query
its HTTP endpoint to confirm that the route you're seeing is also still
there: https://github.com/cloudfoundry/gorouter#instrumentation.
Lastly I would connect to the CCDB and confirm that the app and route are
*not* there. This will reduce the problem to figuring out why Diego isn't
being updated to know that the non-existing app is no longer desired.

On Mon, Apr 4, 2016 at 3:47 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

The route still exists. I was reluctant to delete it and have the
"app" still running. I wanted some way to track it down, not that it has
helped, other than let me know it is still running.

Pushed the app, with a different name/host, with no problems and it
runs as it should.

On Mon, Apr 4, 2016 at 6:17 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Tom,

So you're saying that none of the org/spaces shows the app or the
route, but the app continues to run and be routeable?

I could imagine this happen if some CC Bridge components are not able
to talk to either CC or Diego BBS, leaving the data in the Diego BBS
stale. In the case of stale info, Diego may not know that the LRP is no
longer desired, and it will do the safe thing of keeping it around, and
emitting its route to the gorouter, which just does what it's told (it
doesn't check whether CC knows about the route or not).

Are you able to push new apps or delete other apps with the Diego
backend?

Amit

On Fri, Apr 1, 2016 at 1:00 PM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

JT,

Thanks for responding.

This is a test runtime and small. I checked all orgs and spaces. No
routes matching the app.

Found the route information and the result:

{

"total_results": 0,

"total_pages": 1,

"prev_url": null,

"next_url": null,

"resources": []

}

To learn what the output may look like, I check existing routes with
apps and without. The output appears to be the same as if the app has been
deleted.

Even now, the app url still returns a page from the app, even though
it is deleted.

Thanks,

Tom

On Fri, Apr 1, 2016 at 1:52 PM, JT Archie <jarchie(a)pivotal.io>
wrote:

Tom,

Are you sure the route isn't bound to another application in
another org/space?

When you do `cf routes` it only show routes for the current space.
You can hit specific API endpoints though to get all the apps for a route.

For example, `cf
curl /v2/routes/89fc2a5e-3a9b-4a88-a360-e405cdbd6f87/apps` will show all
the apps for a particular route. Obviously replacing the route ID with the
correct ID. To find that, I recommend going through `CF_TRACE=true cf
routes` and grabbing the ID.

Let see if you can hunt it down that way.

Kind Regards,

JT

On Fri, Apr 1, 2016 at 8:51 AM, Tom Sherrod <tom.sherrod(a)gmail.com>
wrote:

cf 230, diego 0.1450.0, etcd 27, garden-linux 0.330.0
Default to diego true.

Developer deployed a java application. Deleted the application: cf
delete <app> No errors.
The app still responds. The only thing left is the route.
I've not encountered this before. Delete has been delete and even
if route remains, 404 Not Found: Requested route ('<hostname.domain>') does
not exist. is returned.

Pointers on tracking this down appreciated.

Tom