Delivery Status Notification (Failure)


Frank Li <alivedata@...>
 

Hi,

When I run 'bosh deploy' , I got a error''Error 400007: `uaa_z1/0' is not
running after update":

*Started preparing configuration > Binding configuration. Done (00:00:04)*

*Started updating job ha_proxy_z1 > ha_proxy_z1/0. Done (00:00:13)*
*Started updating job nats_z1 > nats_z1/0. Done (00:00:27)*
*Started updating job etcd_z1 > etcd_z1/0. Done (00:00:14)*
*Started updating job postgres_z1 > postgres_z1/0. Done (00:00:22)*
*Started updating job uaa_z1 > uaa_z1/0. Failed: `uaa_z1/0' is not running
after update (00:04:02)*

*Error 400007: `uaa_z1/0' is not running after update*





bosh task 132 --debug

*I, [2015-05-22 03:58:56 #2299] [instance_update(uaa_z1/0)] INFO --
DirectorJobRunner: Waiting for 19.88888888888889 seconds to check uaa_z1/0
status*
*D, [2015-05-22 03:58:56 #2299] [] DEBUG -- DirectorJobRunner: Renewing
lock: lock:deployment:cf-warden*
*D, [2015-05-22 03:59:01 #2299] [] DEBUG -- DirectorJobRunner: Renewing
lock: lock:deployment:cf-warden*
*D, [2015-05-22 03:59:06 #2299] [] DEBUG -- DirectorJobRunner: Renewing
lock: lock:deployment:cf-warden*
*D, [2015-05-22 03:59:11 #2299] [] DEBUG -- DirectorJobRunner: Renewing
lock: lock:deployment:cf-warden*
*I, [2015-05-22 03:59:15 #2299] [instance_update(uaa_z1/0)] INFO --
DirectorJobRunner: Checking if uaa_z1/0 has been updated after
19.88888888888889 seconds*
*D, [2015-05-22 03:59:15 #2299] [instance_update(uaa_z1/0)] DEBUG --
DirectorJobRunner: SENT: agent.04446a2b-a103-4a33-9bbe-d8b07d2c6466
{"method":"get_state","arguments":[],"reply_to":"director.2052649d-bafc-4d7a-8184-caa0373ec71f.55816c88-fea4-45cb-a7a9-13d7579b459a"}*
*D, [2015-05-22 03:59:15 #2299] [] DEBUG -- DirectorJobRunner: RECEIVED:
director.2052649d-bafc-4d7a-8184-caa0373ec71f.55816c88-fea4-45cb-a7a9-13d7579b459a
{"value":{"properties":{"logging":{"max_log_file_size":""}},"job":{"name":"uaa_z1","release":"","template":"uaa","version":"e3278da4c650f21c13cfa935814233bc79f156f0","sha1":"c8f3ee66bd955a58f95dbb7c02ca008c5e91ab6a","blobstore_id":"00e2df47-e90f-414d-8965-f97e1ec81b24","templates":[{"name":"uaa","version":"e3278da4c650f21c13cfa935814233bc79f156f0","sha1":"c8f3ee66bd955a58f95dbb7c02ca008c5e91ab6a","blobstore_id":"00e2df47-e90f-414d-8965-f97e1ec81b24"},{"name":"metron_agent","version":"51cf1a4f2e361bc2a2bbd1bee7fa324fe7029589","sha1":"50fccfa5198b0ccd6b39109ec5585f2502011da3","blobstore_id":"beac8dfd-57e9-45c0-8529-56e4c73154bc"},{"name":"consul_agent","version":"6a3b1fe7963fbcc3dea0eab7db337116ba062056","sha1":"54c6a956f7ee1c906e0f8e8aaac13a25584e7d3f","blobstore_id":"aee73914-cf03-4e7c-98a5-a1695cbc2cc5"}]},"packages":{"common":{"name":"common","version":"99c756b71550530632e393f5189220f170a69647.1","sha1":"6da06edd87b2d78e5e0e9848c26cdafe1b3a94eb","blobstore_id":"6783e7af-2366-4142-7199-ac487f359adb"},"consul":{"name":"consul","version":"d828a4735b02229631673bc9cb6aab8e2d56eda5.1","sha1":"15d541d6f0c8708b9af00f045d58d10951755ad6","blobstore_id":"a9256e97-0940-45dc-6003-77141979c976"},"metron_agent":{"name":"metron_agent","version":"122c9dea1f4be749d48bf1203ed0a407b5a2e1ff.1","sha1":"b8241c6482b03f0d010031e5e99cbae4a909ae05","blobstore_id":"8aa07a49-753a-4200-4cbb-cbb554034986"},"ruby-2.1.4":{"name":"ruby-2.1.4","version":"5a4612011cb6b8338d384acc7802367ae5e11003.1","sha1":"032f58346f55ad468c83e015997ff50091a76ef7","blobstore_id":"afaf9c7a-5633-40cc-7a7a-5d285a560b20"},"uaa":{"name":"uaa","version":"05b84acccba5cb31a170d9cad531d22ccb5df8a5.1","sha1":"ae0a7aa73132db192c2800d0094c607a41d56ddb","blobstore_id":"b474ea8d-5c66-4eea-4a7e-689a0cd0de63"}},"configuration_hash":"c1c40387ae387a29bb69124e3d9f741ee50f0d48","networks":{"cf1":{"cloud_properties":{"name":"random"},"default":["dns","gateway"],"dns_record_name":"0.uaa-z1.cf1.cf-warden.bosh","ip":"10.244.0.130","netmask":"255.255.255.252"}},"resource_pool":{"cloud_properties":{"name":"random"},"name":"medium_z1","stemcell":{"name":"bosh-warden-boshlite-ubuntu-lucid-go_agent","version":"64"}},"deployment":"cf-warden","index":0,"persistent_disk":0,"rendered_templates_archive":{"sha1":"2ebf29eac887fb88dab65aeb911a36403c41b1cb","blobstore_id":"38890fbc-f95e-44a9-9f19-859dc42ec381"},"agent_id":"04446a2b-a103-4a33-9bbe-d8b07d2c6466","bosh_protocol":"1","job_state":"failing","vm":{"name":"755410d0-6697-4505-754e-9521d23788ef"},"ntp":{"message":"file
missing"}}}*
*E, [2015-05-22 03:59:15 #2299] [instance_update(uaa_z1/0)] ERROR --
DirectorJobRunner: Error updating instance:
#<Bosh::Director::AgentJobNotRunning: `uaa_z1/0' is not running after
update>*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/instance_updater.rb:85:in
`update'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:94:in
`block (2 levels) in update_instance'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_formatter.rb:49:in
`with_thread_name'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:92:in
`block in update_instance'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/event_log.rb:97:in
`call'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/event_log.rb:97:in
`advance_and_track'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:91:in
`update_instance'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:85:in
`block (2 levels) in update_instances'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:77:in
`call'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:77:in
`block (2 levels) in create_thread'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:63:in
`loop'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:63:in
`block in create_thread'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`call'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`block in create_with_logging_context'*
*D, [2015-05-22 03:59:15 #2299] [] DEBUG -- DirectorJobRunner: Worker
thread raised exception: `uaa_z1/0' is not running after update -
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/instance_updater.rb:85:in
`update'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:94:in
`block (2 levels) in update_instance'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_formatter.rb:49:in
`with_thread_name'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:92:in
`block in update_instance'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/event_log.rb:97:in
`call'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/event_log.rb:97:in
`advance_and_track'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:91:in
`update_instance'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:85:in
`block (2 levels) in update_instances'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:77:in
`call'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:77:in
`block (2 levels) in create_thread'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:63:in
`loop'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:63:in
`block in create_thread'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`call'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`block in create_with_logging_context'*
*D, [2015-05-22 03:59:16 #2299] [] DEBUG -- DirectorJobRunner: Thread is no
longer needed, cleaning up*
*D, [2015-05-22 03:59:16 #2299] [task:132] DEBUG -- DirectorJobRunner:
Shutting down pool*
*D, [2015-05-22 03:59:16 #2299] [task:132] DEBUG -- DirectorJobRunner:
(0.004399s) SELECT "stemcells".* FROM "stemcells" INNER JOIN
"deployments_stemcells" ON (("deployments_stemcells"."stemcell_id" =
"stemcells"."id") AND ("deployments_stemcells"."deployment_id" = 1))*
*D, [2015-05-22 03:59:16 #2299] [task:132] DEBUG -- DirectorJobRunner:
Deleting lock: lock:deployment:cf-warden*
*D, [2015-05-22 03:59:16 #2299] [] DEBUG -- DirectorJobRunner: Lock renewal
thread exiting*
*D, [2015-05-22 03:59:16 #2299] [task:132] DEBUG -- DirectorJobRunner:
Deleted lock: lock:deployment:cf-warden*
*I, [2015-05-22 03:59:16 #2299] [task:132] INFO -- DirectorJobRunner:
sending update deployment error event*
*D, [2015-05-22 03:59:16 #2299] [task:132] DEBUG -- DirectorJobRunner:
SENT: hm.director.alert
{"id":"7245631b-b6b3-43df-bd43-65b19e23f6ae","severity":3,"title":"director
- error during update deployment","summary":"Error during update deployment
for cf-warden against Director c6f166bd-ddac-4f7d-9c57-d11c6ad5133b:
#<Bosh::Director::AgentJobNotRunning: `uaa_z1/0' is not running after
update>","created_at":1432267156}*
*E, [2015-05-22 03:59:16 #2299] [task:132] ERROR -- DirectorJobRunner:
`uaa_z1/0' is not running after update*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/instance_updater.rb:85:in
`update'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:94:in
`block (2 levels) in update_instance'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_formatter.rb:49:in
`with_thread_name'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:92:in
`block in update_instance'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/event_log.rb:97:in
`call'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/event_log.rb:97:in
`advance_and_track'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:91:in
`update_instance'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2811.0/lib/bosh/director/job_updater.rb:85:in
`block (2 levels) in update_instances'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:77:in
`call'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:77:in
`block (2 levels) in create_thread'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:63:in
`loop'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh_common-1.2811.0/lib/common/thread_pool.rb:63:in
`block in create_thread'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`call'*
*/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in
`block in create_with_logging_context'*
*D, [2015-05-22 03:59:16 #2299] [task:132] DEBUG -- DirectorJobRunner:
(0.000396s) BEGIN*
*D, [2015-05-22 03:59:16 #2299] [task:132] DEBUG -- DirectorJobRunner:
(0.001524s) UPDATE "tasks" SET "state" = 'error', "timestamp" = '2015-05-22
03:59:16.090280+0000', "description" = 'create deployment', "result" =
'`uaa_z1/0'' is not running after update', "output" =
'/var/vcap/store/director/tasks/132', "checkpoint_time" = '2015-05-22
03:58:52.002311+0000', "type" = 'update_deployment', "username" = 'admin'
WHERE ("id" = 132)*
*D, [2015-05-22 03:59:16 #2299] [task:132] DEBUG -- DirectorJobRunner:
(0.002034s) COMMIT*
*I, [2015-05-22 03:59:16 #2299] [] INFO -- DirectorJobRunner: Task took 5
minutes 55.32297424799998 seconds to process.*





uaa section in cf-manifest.yml as following:

*uaa:*
*admin:*
*client_secret: admin-secret*
*authentication:*
*policy:*
*countFailuresWithinSeconds: null*
*lockoutAfterFailures: null*
*lockoutPeriodSeconds: null*
*batch:*
*password: batch-password*
*username: batch-username*
*catalina_opts: -Xmx192m -XX:MaxPermSize=128m*
*cc:*
*client_secret: cc-secret*
*clients:*
*app-direct:*
*access-token-validity: 1209600*
*authorities: app_direct_invoice.write*
*authorized-grant-types:
authorization_code,client_credentials,password,refresh_token,implicit*
*override: true*
*redirect-uri: https://console.10.244.0.34.xip.io
<https://console.10.244.0.34.xip.io/>*
*refresh-token-validity: 1209600*
*secret: app-direct-secret*
*cc-service-dashboards:*
*authorities: clients.read,clients.write,clients.admin*
*authorized-grant-types: client_credentials*
*scope: openid,cloud_controller_service_permissions.read*
*secret: cc-broker-secret*
*cloud_controller_username_lookup:*
*authorities: scim.userids*
*authorized-grant-types: client_credentials*
*secret: cloud-controller-username-lookup-secret*
*developer_console:*
*access-token-validity: 1209600*
*authorities:
scim.write,scim.read,cloud_controller.read,cloud_controller.write,password.write,uaa.admin,uaa.resource,cloud_controller.admin,billing.admin*
*authorized-grant-types: authorization_code,client_credentials*
*override: true*
*redirect-uri: https://console.10.244.0.34.xip.io/oauth/callback
<https://console.10.244.0.34.xip.io/oauth/callback>*
*refresh-token-validity: 1209600*
*scope:
openid,cloud_controller.read,cloud_controller.write,password.write,console.admin,console.support*
*secret: console-secret*
*doppler:*
*authorities: uaa.resource*
*override: true*
*secret: doppler-secret*
*gorouter:*
*authorities:
clients.read,clients.write,clients.admin,route.admin,route.advertise*
*authorized-grant-types: client_credentials,refresh_token*
*scope: openid,cloud_controller_service_permissions.read*
*secret: gorouter-secret*
*login:*
*authorities:
oauth.login,scim.write,clients.read,notifications.write,critical_notifications.write,emails.write,scim.userids,password.write*
*authorized-grant-types:
authorization_code,client_credentials,refresh_token*
*override: true*
*redirect-uri: http://login.10.244.0.34.xip.io
<http://login.10.244.0.34.xip.io/>*
*scope: openid,oauth.approvals*
*secret: login-secret*
*notifications:*
*authorities: cloud_controller.admin,scim.read*
*authorized-grant-types: client_credentials*
*secret: notification-secret*
*issuer: https://uaa.10.244.0.34.xip.io <https://uaa.10.244.0.34.xip.io/>*
*jwt:*
*signing_key: |+*
*-----BEGIN RSA PRIVATE KEY-----*
*MIICXAIBAAKBgQDHFr+KICms+tuT1OXJwhCUmR2dKVy7psa8xzElSyzqx7oJyfJ1*
*JZyOzToj9T5SfTIq396agbHJWVfYphNahvZ/7uMXqHxf+ZH9BL1gk9Y6kCnbM5R6*
*0gfwjyW1/dQPjOzn9N394zd2FJoFHwdq9Qs0wBugspULZVNRxq7veq/fzwIDAQAB*
*AoGBAJ8dRTQFhIllbHx4GLbpTQsWXJ6w4hZvskJKCLM/o8R4n+0W45pQ1xEiYKdA*
*Z/DRcnjltylRImBD8XuLL8iYOQSZXNMb1h3g5/UGbUXLmCgQLOUUlnYt34QOQm+0*
*KvUqfMSFBbKMsYBAoQmNdTHBaz3dZa8ON9hh/f5TT8u0OWNRAkEA5opzsIXv+52J*
*duc1VGyX3SwlxiE2dStW8wZqGiuLH142n6MKnkLU4ctNLiclw6BZePXFZYIK+AkE*
*xQ+k16je5QJBAN0TIKMPWIbbHVr5rkdUqOyezlFFWYOwnMmw/BKa1d3zp54VP/P8*
*+5aQ2d4sMoKEOfdWH7UqMe3FszfYFvSu5KMCQFMYeFaaEEP7Jn8rGzfQ5HQd44ek*
*lQJqmq6CE2BXbY/i34FuvPcKU70HEEygY6Y9d8J3o6zQ0K9SYNu+pcXt4lkCQA3h*
*jJQQe5uEGJTExqed7jllQ0khFJzLMx0K6tj0NeeIzAaGCQz13oo2sCdeGRHO4aDh*
*HH6Qlq/6UOV5wP8+GAcCQFgRCcB+hrje8hfEEefHcFpyKH+5g1Eu1k0mLrxK2zd+*
*4SlotYRHgPCEubokb2S1zfZDWIXW3HmggnGgM949TlY=*
*-----END RSA PRIVATE KEY-----*
*verification_key: |+*
*-----BEGIN PUBLIC KEY-----*
*MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDHFr+KICms+tuT1OXJwhCUmR2d*
*KVy7psa8xzElSyzqx7oJyfJ1JZyOzToj9T5SfTIq396agbHJWVfYphNahvZ/7uMX*
*qHxf+ZH9BL1gk9Y6kCnbM5R60gfwjyW1/dQPjOzn9N394zd2FJoFHwdq9Qs0wBug*
*spULZVNRxq7veq/fzwIDAQAB*
*-----END PUBLIC KEY-----*
*ldap: null*
*login: null*
*no_ssl: true*
*restricted_ips_regex:
10\.\d{1,3}\.\d{1,3}\.\d{1,3}|192\.168\.\d{1,3}\.\d{1,3}|169\.254\.\d{1,3}\.\d{1,3}|127\.\d{1,3}\.\d{1,3}\.\d{1,3}|172\.1[6-9]{1}\.\d{1,3}\.\d{1,3}|172\.2[0-9]{1}\.\d{1,3}\.\d{1,3}|172\.3[0-1]{1}\.\d{1,3}\.\d{1,3}*
*scim:*
*external_groups: null*
*userids_enabled: true*
*users:*
*-
admin|admin|scim.write,scim.read,openid,cloud_controller.admin,clients.read,clients.write,doppler.firehose*
*spring_profiles: null*
*url: https://uaa.10.244.0.34.xip.io <https://uaa.10.244.0.34.xip.io/>*
*user: null*
*uaadb:*
*address: 10.244.0.30*
*databases:*
*- citext: true*
*name: uaadb*
*tag: uaa*
*db_scheme: postgresql*
*port: 5524*
*roles:*
*- name: uaaadmin*
*password: admin*
*tag: admin*



Can anyone help me ?Thanks!


Best Regards,

Frank

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.