The issue was likely the disk. The staging_disk_limit_mb tells the cloud
controller how much disk space is needed for staging an app (unless the app
asks for more). This is normally much higher than the size of the app
itself because you have to accommodate all of the buildpacks on the DEA in
addition to the app itself.

So the staging_disk_limit_mb was 6144 mb, but the DEA was set to advertise
2048 mb.

Joseph Palermo
CF Runtime Team

On Thu, Jun 4, 2015 at 9:12 PM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

Finally, it works after I changed the configuration related to memory and
disk for dea.

< disk_mb: 2048
---

disk_mb: 10000

1025c1029
< memory_overcommit_factor: 3
---

memory_overcommit_factor: 8

1028c1032
< staging_disk_limit_mb: 6144
---

staging_disk_limit_mb: 4096

[#1] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8192,"available_disk":20000}'
[#2] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8192,"available_disk":20000}'

After I deployed a simple php-demo application, they are changed to

[#8] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8064,"available_disk":18976}'
[#9] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8064,"available_disk":18976}'


However, I still cannot understand why my previous configuration led to
"Staging error: no available stagers" as the nats messages below said it
has enough resource. My php application is only consuming 128M memory and
1G disk. Who can share some insights?

[#41] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'


ubuntu(a)boshclivm:~/apps/cf-php-demo$ cat manifest.yml
---
applications:
- name: cf-php-demo
memory: 128M
instances: 1
host: cf-php-demo
path: .

ubuntu(a)boshclivm:~/apps/cf-php-demo$ cf apps
name requested state instances memory disk urls
cf-php-demo started 1/1 128M 1G
cf-php-demo.runmyapp.io



On Thu, Jun 4, 2015 at 5:08 PM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

I got the nats message on 'staging.advertise'. It has the enough
resource. but it seems not correct. And it also cannot explain the error -
Server error, status code: 400, error code: 170001, message: Staging error:
no available stagers.

[#41] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#42] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#43] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#44] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'



+------------------------------------+---------+---------------+---------------+
| Job/index | State | Resource Pool |
IPs |

+------------------------------------+---------+---------------+---------------+
| api_worker_z1/0 | running | small_z1 |
100.64.0.23 |
| api_z1/0 | running | medium_z1 |
100.64.0.21 |
| clock_global/0 | running | medium_z1 |
100.64.0.22 |
| etcd_z1/0 | running | medium_z1 |
100.64.1.8 |
| ha_proxy_z1/0 | running | router_z1 |
100.64.1.0 |
| | | |
137.172.74.90 |
| hm9000_z1/0 | running | medium_z1 |
100.64.0.24 |
| loggregator_trafficcontroller_z1/0 | running | small_z1 |
100.64.0.27 |
| loggregator_z1/0 | running | medium_z1 |
100.64.0.26 |
| login_z1/0 | running | medium_z1 |
100.64.0.20 |
| nats_z1/0 | running | medium_z1 |
100.64.1.2 |
| nfs_z1/0 | running | medium_z1 |
100.64.1.3 |
| postgres_z1/0 | running | medium_z1 |
100.64.1.4 |
| router_z1/0 | running | router_z1 |
100.64.1.5 |
| runner_z1/0 | running | runner_z1 |
100.64.0.25 |
| stats_z1/0 | running | small_z1 |
100.64.0.18 |
| uaa_z1/0 | running | medium_z1 |
100.64.0.19 |

+------------------------------------+---------+---------------+---------------+


- 100.64.0.25

m1.large | 8GB RAM | 4 VCPU | 20.0GB Disk

92cf66ec-f2e1-4505-bd25-28c02e991535 | m1.large | 8192 |
20 | 20 | | 4 | 1.0 | True


On Thu, Jun 4, 2015 at 11:57 AM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

From the source code
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/app_stager_task.rb:26,
it seems there is no enough for memory or disk.

def stage(&completion_callback)
@stager_id = @stager_pool.find_stager(@app.stack.name,
staging_task_memory_mb, staging_task_disk_mb)
raise Errors::ApiError.new_from_details('StagingError', 'no
available stagers') unless @stager_id


However, this is my first app. It should be light. The DEA is using
m1.large which is
m1.large | 4096 | 20

Anyone has the same error? and any suggestion on manifest or debug tips?

Another question, I want to add more debug information in
cloud_controller_ng.log. I tried to add some code in
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/app_stager_task.rb,
but it did not show in the log. How to do?


On Thu, Jun 4, 2015 at 10:14 AM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

attached the deployment manifest. This is generated by spiff and then I
modified it.

On Thu, Jun 4, 2015 at 12:47 AM, Takeshi Morikawa <moog0814(a)gmail.com>
wrote:

Please check the 'staging.advertise' of nats message
https://github.com/cloudfoundry/dea_ng#staging

sample command:
bundle exec nats-sub -s
nats://[nats.user]:[nats.password]@[nats_ipaddress]:[nats.port]
'staging.advertise'


I have one additional request
Can you share your bosh deployment manifest?

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

I am sorry that I was no help to you

2015-06-05 13:12 GMT+09:00 Guangcai Wang <guangcai.wang(a)gmail.com>:

Finally, it works after I changed the configuration related to memory and
disk for dea.

< disk_mb: 2048
---

disk_mb: 10000

1025c1029
< memory_overcommit_factor: 3
---

memory_overcommit_factor: 8

1028c1032
< staging_disk_limit_mb: 6144
---

staging_disk_limit_mb: 4096

[#1] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8192,"available_disk":20000}'
[#2] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8192,"available_disk":20000}'

After I deployed a simple php-demo application, they are changed to

[#8] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8064,"available_disk":18976}'
[#9] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8064,"available_disk":18976}'


However, I still cannot understand why my previous configuration led to
"Staging error: no available stagers" as the nats messages below said it
has enough resource. My php application is only consuming 128M memory and
1G disk. Who can share some insights?

[#41] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'


ubuntu(a)boshclivm:~/apps/cf-php-demo$ cat manifest.yml
---
applications:
- name: cf-php-demo
memory: 128M
instances: 1
host: cf-php-demo
path: .

ubuntu(a)boshclivm:~/apps/cf-php-demo$ cf apps
name requested state instances memory disk urls
cf-php-demo started 1/1 128M 1G
cf-php-demo.runmyapp.io



On Thu, Jun 4, 2015 at 5:08 PM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

I got the nats message on 'staging.advertise'. It has the enough
resource. but it seems not correct. And it also cannot explain the error -
Server error, status code: 400, error code: 170001, message: Staging error:
no available stagers.

[#41] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#42] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#43] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#44] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'



+------------------------------------+---------+---------------+---------------+
| Job/index | State | Resource Pool |
IPs |

+------------------------------------+---------+---------------+---------------+
| api_worker_z1/0 | running | small_z1 |
100.64.0.23 |
| api_z1/0 | running | medium_z1 |
100.64.0.21 |
| clock_global/0 | running | medium_z1 |
100.64.0.22 |
| etcd_z1/0 | running | medium_z1 |
100.64.1.8 |
| ha_proxy_z1/0 | running | router_z1 |
100.64.1.0 |
| | | |
137.172.74.90 |
| hm9000_z1/0 | running | medium_z1 |
100.64.0.24 |
| loggregator_trafficcontroller_z1/0 | running | small_z1 |
100.64.0.27 |
| loggregator_z1/0 | running | medium_z1 |
100.64.0.26 |
| login_z1/0 | running | medium_z1 |
100.64.0.20 |
| nats_z1/0 | running | medium_z1 |
100.64.1.2 |
| nfs_z1/0 | running | medium_z1 |
100.64.1.3 |
| postgres_z1/0 | running | medium_z1 |
100.64.1.4 |
| router_z1/0 | running | router_z1 |
100.64.1.5 |
| runner_z1/0 | running | runner_z1 |
100.64.0.25 |
| stats_z1/0 | running | small_z1 |
100.64.0.18 |
| uaa_z1/0 | running | medium_z1 |
100.64.0.19 |

+------------------------------------+---------+---------------+---------------+


- 100.64.0.25

m1.large | 8GB RAM | 4 VCPU | 20.0GB Disk

92cf66ec-f2e1-4505-bd25-28c02e991535 | m1.large | 8192 |
20 | 20 | | 4 | 1.0 | True


On Thu, Jun 4, 2015 at 11:57 AM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

From the source code
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/app_stager_task.rb:26,
it seems there is no enough for memory or disk.

def stage(&completion_callback)
@stager_id = @stager_pool.find_stager(@app.stack.name,
staging_task_memory_mb, staging_task_disk_mb)
raise Errors::ApiError.new_from_details('StagingError', 'no
available stagers') unless @stager_id


However, this is my first app. It should be light. The DEA is using
m1.large which is
m1.large | 4096 | 20

Anyone has the same error? and any suggestion on manifest or debug tips?

Another question, I want to add more debug information in
cloud_controller_ng.log. I tried to add some code in
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/app_stager_task.rb,
but it did not show in the log. How to do?


On Thu, Jun 4, 2015 at 10:14 AM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

attached the deployment manifest. This is generated by spiff and then I
modified it.

On Thu, Jun 4, 2015 at 12:47 AM, Takeshi Morikawa <moog0814(a)gmail.com>
wrote:

Please check the 'staging.advertise' of nats message
https://github.com/cloudfoundry/dea_ng#staging

sample command:
bundle exec nats-sub -s
nats://[nats.user]:[nats.password]@[nats_ipaddress]:[nats.port]
'staging.advertise'


I have one additional request
Can you share your bosh deployment manifest?

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

debug log sample code

https://github.com/nttlabs/cloud_controller_ng/blob/zone-extention/lib/cloud_controller/stager/stager_pool.rb#L35
https://github.com/nttlabs/cloud_controller_ng/blob/zone-extention/lib/cloud_controller/stager/stager_pool.rb#L28-L30


2015-06-04 10:57 GMT+09:00 Guangcai Wang <guangcai.wang(a)gmail.com>:

I got the nats message on 'staging.advertise'. It has the enough resource.
but it seems not correct. And it also cannot explain the error - Server
error, status code: 400, error code: 170001, message: Staging error: no
available stagers.

[#41] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#42] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#43] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#44] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'


+------------------------------------+---------+---------------+---------------+
| Job/index | State | Resource Pool |
IPs |
+------------------------------------+---------+---------------+---------------+
| api_worker_z1/0 | running | small_z1 |
100.64.0.23 |
| api_z1/0 | running | medium_z1 |
100.64.0.21 |
| clock_global/0 | running | medium_z1 |
100.64.0.22 |
| etcd_z1/0 | running | medium_z1 |
100.64.1.8 |
| ha_proxy_z1/0 | running | router_z1 |
100.64.1.0 |
| | | |
137.172.74.90 |
| hm9000_z1/0 | running | medium_z1 |
100.64.0.24 |
| loggregator_trafficcontroller_z1/0 | running | small_z1 |
100.64.0.27 |
| loggregator_z1/0 | running | medium_z1 |
100.64.0.26 |
| login_z1/0 | running | medium_z1 |
100.64.0.20 |
| nats_z1/0 | running | medium_z1 |
100.64.1.2 |
| nfs_z1/0 | running | medium_z1 |
100.64.1.3 |
| postgres_z1/0 | running | medium_z1 |
100.64.1.4 |
| router_z1/0 | running | router_z1 |
100.64.1.5 |
| runner_z1/0 | running | runner_z1 |
100.64.0.25 |
| stats_z1/0 | running | small_z1 |
100.64.0.18 |
| uaa_z1/0 | running | medium_z1 |
100.64.0.19 |
+------------------------------------+---------+---------------+---------------+


- 100.64.0.25

m1.large | 8GB RAM | 4 VCPU | 20.0GB Disk

92cf66ec-f2e1-4505-bd25-28c02e991535 | m1.large | 8192 | 20
| 20 | | 4 | 1.0 | True


On Thu, Jun 4, 2015 at 11:57 AM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

From the source code
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/app_stager_task.rb:26,
it seems there is no enough for memory or disk.

def stage(&completion_callback)
@stager_id = @stager_pool.find_stager(@app.stack.name,
staging_task_memory_mb, staging_task_disk_mb)
raise Errors::ApiError.new_from_details('StagingError', 'no
available stagers') unless @stager_id


However, this is my first app. It should be light. The DEA is using
m1.large which is
m1.large | 4096 | 20

Anyone has the same error? and any suggestion on manifest or debug tips?

Another question, I want to add more debug information in
cloud_controller_ng.log. I tried to add some code in
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/app_stager_task.rb,
but it did not show in the log. How to do?


On Thu, Jun 4, 2015 at 10:14 AM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

attached the deployment manifest. This is generated by spiff and then I
modified it.

On Thu, Jun 4, 2015 at 12:47 AM, Takeshi Morikawa <moog0814(a)gmail.com>
wrote:

Please check the 'staging.advertise' of nats message
https://github.com/cloudfoundry/dea_ng#staging

sample command:
bundle exec nats-sub -s
nats://[nats.user]:[nats.password]@[nats_ipaddress]:[nats.port]
'staging.advertise'


I have one additional request
Can you share your bosh deployment manifest?

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

From the source code
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/app_stager_task.rb:26,
it seems there is no enough for memory or disk.

def stage(&completion_callback)
@stager_id = @stager_pool.find_stager(@app.stack.name,
staging_task_memory_mb, staging_task_disk_mb)
raise Errors::ApiError.new_from_details('StagingError', 'no
available stagers') unless @stager_id


However, this is my first app. It should be light. The DEA is using
m1.large which is
m1.large | 4096 | 20

Anyone has the same error? and any suggestion on manifest or debug tips?

Another question, I want to add more debug information in
cloud_controller_ng.log. I tried to add some code in
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/app_stager_task.rb,
but it did not show in the log. How to do?


On Thu, Jun 4, 2015 at 10:14 AM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:

attached the deployment manifest. This is generated by spiff and then I
modified it.

On Thu, Jun 4, 2015 at 12:47 AM, Takeshi Morikawa <moog0814(a)gmail.com>
wrote:

Please check the 'staging.advertise' of nats message
https://github.com/cloudfoundry/dea_ng#staging

sample command:
bundle exec nats-sub -s
nats://[nats.user]:[nats.password]@[nats_ipaddress]:[nats.port]
'staging.advertise'


I have one additional request
Can you share your bosh deployment manifest?

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

attached the deployment manifest. This is generated by spiff and then I
modified it.

On Thu, Jun 4, 2015 at 12:47 AM, Takeshi Morikawa <moog0814(a)gmail.com>
wrote:

Please check the 'staging.advertise' of nats message
https://github.com/cloudfoundry/dea_ng#staging

sample command:
bundle exec nats-sub -s
nats://[nats.user]:[nats.password]@[nats_ipaddress]:[nats.port]
'staging.advertise'


I have one additional request
Can you share your bosh deployment manifest?

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

attached the deployment manifest. cf-deployment-single-az.yml
<http://cf-dev.70369.x6.nabble.com/file/n290/cf-deployment-single-az.yml>



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-Staging-error-no-available-stagers-status-code-400-error-code-170001-tp271p290.html
Sent from the CF Dev mailing list archive at Nabble.com.

Please check the 'staging.advertise' of nats message
https://github.com/cloudfoundry/dea_ng#staging

sample command:
bundle exec nats-sub -s
nats://[nats.user]:[nats.password]@[nats_ipaddress]:[nats.port]
'staging.advertise'


I have one additional request
Can you share your bosh deployment manifest?

Resend the question.

I deployed the cf into openstack successfully. However, I got failure when
I tried to push my first php example.

Starting app cf-php-demo in org system / space dev as admin...
FAILED
Server error, status code: 400, error code: 170001, message: Staging error:
no available stagers


Below is my findings. The DEA and CC have enough resources (4G RAM, 20G
disk). Any clues
to debug? Attached debug.log. debug.log
<http://cf-dev.70369.x6.nabble.com/file/n269/debug.log

*My env:*
BOSH 1.2978.0
cf version 6.10.0-b78bf10-2015-02-11T22:26:40+00:00
cf release: 207
stemcell: bosh-openstack-kvm-ubuntu-trusty-go_agent | 2969


ubuntu(a)boshclivm:~/apps/cf-php-demo$ CF_TRACE=debug.log cf push
Using manifest file /home/ubuntu/apps/cf-php-demo/manifest.yml

Creating app cf-php-demo in org system / space dev as admin...
OK

Using route cf-php-demo.runmyapp.io
Binding cf-php-demo.runmyapp.io to cf-php-demo...
OK

Uploading cf-php-demo...
Uploading app files from: /home/ubuntu/apps/cf-php-demo
Uploading 231.9K, 13 files
Done uploading
OK

Starting app cf-php-demo in org system / space dev as admin...
FAILED
Server error, status code: 400, error code: 170001, message: Staging error:
no available stagers


ubuntu(a)boshclivm:~/apps/cf-php-demo$ cf apps
Getting apps in org system / space dev as admin...
OK

name requested state instances memory disk urls
cf-php-demo started 0/1 128M 1G
cf-php-demo.cfapps.io
ubuntu(a)boshclivm:~/apps/cf-php-demo$


Additional error messages:
1. nginx.access.log
api.au.apaas.com - [03/Jun/2015:06:19:51 +0000] "PUT
/v2/apps/7b417005-6716-4d5e-bec2-246a51b588c6?async=true&inline-relations-depth=1
HTTP/1.1" 400 435 "-" "go-cli 6.10.0-b78bf10 / linux" 137.172.74.86,
100.64.1.0, 100.64.1.5
vcap_request_id:ae51df28-8371-44f2-4d18-5e13ded3a467::6faf587a-aba8-4467-8e1e-75b018eff93b
response_time:0.297

2. cloud_controller_ng.log
{"timestamp":1433313002.4425669,"message":"Request failed: 400:
{\"code\"=>170001, \"description\"=>\"Staging error: no available stagers\",
\"error_code\"=>\"CF-StagingError\",
\"backtrace\"=>[\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/app_stager_task.rb:26:in
`stage'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/stager.rb:45:in
`stage_app'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_observer.rb:57:in
`react_to_state_change'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/app_observer.rb:27:in
`updated'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/app.rb:534:in
`after_commit'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.21.0/lib/sequel/model/base.rb:1920:in
`block in _save'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.21.0/lib/sequel/database/transactions.rb:280:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.21.0/lib/sequel/database/transactions.rb:280:in
`block in remove_transaction'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.21.0/lib/sequel/database/transactions.rb:280:in
`each'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.21.0/lib/sequel/database/transactions.rb:280:in
`remove_transaction'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.21.0/lib/sequel/database/transactions.rb:156:in
`_transaction'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.21.0/lib/sequel/database/transactions.rb:108:in
`block in transaction'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.21.0/lib/sequel/database/connecting.rb:250:in
`block in synchronize'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.21.0/lib/sequel/connection_pool/threaded.rb:98:in
`hold'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.21.0/lib/sequel/database/connecting.rb:250:in
`synchronize'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.21.0/lib/sequel/database/transactions.rb:97:in
`transaction'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/model_controller.rb:64:in
`update'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/base_controller.rb:76:in
`dispatch'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/rest_controller/routes.rb:16:in
`block in define_route'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1602:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1602:in
`block in compile!'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in
`[]'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in
`block (3 levels) in route!'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:985:in
`route_eval'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in
`block (2 levels) in route!'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1006:in
`block in process_route'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in
`catch'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in
`process_route'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:964:in
`block in route!'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in
`each'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in
`route!'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in
`block in dispatch!'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`block in invoke'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`catch'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`invoke'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in
`dispatch!'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in
`block in call!'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`block in invoke'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`catch'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`invoke'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in
`call!'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/nulllogger.rb:9:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/head.rb:11:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/builder.rb:138:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/urlmap.rb:65:in
`block in call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in
`each'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/urlmap.rb:50:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/rack-1.5.2/lib/rack/builder.rb:138:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/thin-1.6.3/lib/thin/connection.rb:86:in
`block in pre_process'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/thin-1.6.3/lib/thin/connection.rb:84:in
`catch'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/thin-1.6.3/lib/thin/connection.rb:84:in
`pre_process'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in
`call'\",
\"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:1037:in
`block in
spawn_threadpool'\"]}","log_level":"info","source":"cc.api","data":{"request_guid":"d916023a-d364-492b-50bb-624e5862e455::2c136550-80cb-4d5b-8ab3-6c2dd397fc1a"},"thread_id":69941342398660,"fiber_id":69941329029340,"process_id":1816,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/sinatra/vcap.rb","lineno":51,"method":"block
in registered"}