See inline...

On Fri, Oct 28, 2016 at 10:48 AM, Anuj Jain <anuj17280(a)gmail.com> wrote:

Hi Daniel,

I tried starting the application with CF_TRACE enabled and/or checked
the cc logs but did not find any thing useful yet

in CF Trace only seeing generic error msg eg: 'Stager error: staging
failed: staging failed' and same with cc logs too - check below for your
reference.

Also checked bosh instances --ps (all bosh instances running all the
micro services) - (As mentioned earlier - I am only seeing this issue when
trying to run without giving buildpack information manually -b option -
which suggest diego app is not auto-detecting JAVA buildpack)

Noted. It is an interesting point that this is isolated to just those
circumstances. I don't think that this is a build pack error because
you're not getting to the point where the application stages (which is
where the build pack runs).

Starting app test01 in org tests / space dev as admin...

REQUEST: [2016-10-28T07:52:35-04:00]
PUT /v2/apps/2ceabe54-85f1-4b86-a544-850cd77d1456?async=true&inline-relations-depth=1
HTTP/1.1
Host: api.example.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Connection: close
Content-Type: application/json
User-Agent: go-cli 6.22.1+6b7af9c / linux

{"state":"STARTED"}

RESPONSE: [2016-10-28T07:52:47-04:00]
HTTP/1.1 500 Internal Server Error
Connection: close
Content-Length: 120
Content-Type: application/json;charset=utf-8
Date: Fri, 28 Oct 2016 11:52:46 GMT
Server: nginx
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 5c08c485-e88e-4f2d-4187-870142089a10
X-Vcap-Request-Id: 7797265c-adb3-440c-506c-0b2e11bc82fa::d52d2ae9-7d22-
4bd7-8984-8f82fbd8c76b

{
"code": 170011,
"description": "Stager error: staging failed: staging failed",
"error_code": "CF-StagerError"
}

This is the request that's failing. Try looking for the request guids
listed above in your logs and see if you can use them to track down the
error. There's almost always a stack trace when you see a 500 error. It
can be a little tricky to spot though because they are mashed up into the
JSON log format. I think looking for the request guid should help.

FAILED
Server error, status code: 500, error code: 170011, message: Stager error:
staging failed: staging failed
============================================================
==================================

========================================= CC logs
===============================================
{"timestamp":1477662410.5205755,"message":"stage.
response","log_level":"info","source":"cc.stager.client"
,"data":{"request_guid":"bb34af20-833d-43c5-4e69-
aee81080e4cc::72f15613-7113-407d-8c11-b29f69939b2e","sta
ging_guid":"e11b496e-a04d-4ea8-9baf-e1bdd34a0aa1-
c9289808e4f94936a815ac7d694ac684","response_code":"500"}
,"thread_id":46950135289800,"fiber_id":69867206190260,"
process_id":17850,"file":"/var/vcap/data/packages/
cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-
3849c951dd68411839376f0fb70815b2e8a87213/c
loud_controller_ng/lib/cloud_controller/diego/stager_
client.rb","lineno":29,"method":"stage"}
{"timestamp":1477662410.5209706,"message":"stage.app",
"log_level":"error","source":"cc.stager.client","data":{"
request_guid":"bb34af20-833d-43c5-4e69-aee81080e4cc::
72f15613-7113-407d-8c11-b29f69939b2e","staging_guid":"
e11b496e-a04d-4ea8-9baf-e1bdd34a0aa1-c9289808e4f94936a815ac7d694ac6
84","error":{"details":{"details_hash":{"http_code":500,"message":"Stager
error: %s","code":170011},"name":"StagerError"},"args":["staging failed:
staging failed"]}},"thread_id":46950135289800,"fiber_id":
69867206190260,"process_id":17850,"file":"/var/vcap/data/
packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-
3849c951dd68411839376f0fb70815b2e8a87213/cloud_controller_
ng/lib/cloud_controller/diego/stager.rb","lineno":22,"method":"rescue in
stage"}


{"timestamp":1477662410.521489,"message":"diego.
staging.finished","log_level":"info","source":"cc.stager","
data":{"request_guid":"bb34af20-833d-43c5-4e69-
aee81080e4cc::72f15613-7113-407d-8c11-b29f69939b2e","
response":{"error":{"id":"StagingError","message":"Stager error: staging
failed: staging failed"}}},"thread_id":46950135289800,"fiber_id":
69867206190260,"process_id":17850,"file":"/var/vcap/data/
packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-
3849c951dd68411839376f0fb70815b2e8a87213/cloud_controller_
ng/lib/cloud_controller/diego/staging_completion_handler_
base.rb","lineno":13,"method":"staging_complete"}

These look like an issue with the code that is talking to Diego. You might
want to look through the logs for the CC Bridge components (in particular,
stager) and see if there's anything that stands out there.

https://github.com/cloudfoundry/diego-design-notes#what-are-all-these-repos-and-what-do-they-do

Dan

============================================================
============================

Note: I have changed the domain name to example.com



On Thu, Oct 27, 2016 at 9:55 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:

The 500 error is coming from Cloud Controller, which indicates a problem
there. Check out the CC logs and look for a stack trace to get more
details on what is failing. It doesn't mean that CC is necessarily at
fault, CC depends on other components so one of those could be failing, but
it is a good place to start looking.

Also, run `bosh instances --ps` and see if any report failing. That
might be another way to spot the problem.

Dan


On Thu, Oct 27, 2016 at 8:42 AM, Anuj Jain <anuj17280(a)gmail.com> wrote:

Hi,



I am facing issue while trying to deploy application with diego
enabled - I need to provide the build pack information to make app run with
diego cell - seems if want to run the application on diego cell, we need to
pass/provide build pack information explicitly.


Belwo are the steps which I am executing to run the app on diego cell -
which is not working.

1. cf push diego-test-app -p ./sample.war --no-start
2. cf enable-diego diego-test-app
3. cf start diego-test-app


Error: Server error, status code: 500, error code: 170011, message:
Stager error: staging failed: staging failed


If I am trying to run the application with JAVA build pack information -
I can run the application.


1. cf push diego-test-app -p ./sample.war -b java_buildpack
--no-start
2. cf enable-diego diego-test-app
3. cf start diego-test-app

Note: I am not seeing this issue when trying to run the application on
dea runner's - cloud foundry cli is detecting buildpack automatically.


============ My Cloud Foundry stemcells and release version information
==============


$ bosh stemcells
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
Acting as user 'admin' on 'my-bosh'

+------------------------------------------+---------------+
---------+-----------------------------------------+
| Name | OS | Version |
CID |
+------------------------------------------+---------------+
---------+-----------------------------------------+
| bosh-vsphere-esxi-ubuntu-trusty-go_agent | ubuntu-trusty | 3197* |
sc-b53d78ad-1bbc-4c3a-8656-aed0b7208ba7 |
+------------------------------------------+---------------+
---------+-----------------------------------------+

(*) Currently in-use

Stemcells total: 1


$ bosh releases
RSA 1024 bit CA certificates are loaded due to old openssl compatibility
Acting as user 'admin' on 'my-bosh'

+--------------------+-----------+-------------+
| Name | Versions | Commit Hash |
+--------------------+-----------+-------------+
| cf | 230 | 3de108d9+ |
| | 231* | f67ebb73+ |
| | 232 | 67a5f78e+ |
| chef_client | 2* | 00000000 |
| diego | 0.1454.0* | 72b65532 |
| diego-docker-cache | 0.1021.0 | 772f7d33+ |
| etcd | 36* | 993cbd05 |
| garden-linux | 0.333.0* | 98b28e0d |
+--------------------+-----------+-------------+
(*) Currently deployed
(+) Uncommitted changes

Releases total: 6


==========================================================


- Anuj