Hi,
I just deployed v233 in a vSphere environment. Both api_z1 and api_z2 are failing. Any help would be appreciated.
Stemcell version:
| bosh-vsphere-esxi-ubuntu-trusty-go_agent | bosh-vsphere-esxi-ubuntu-trusty-go_agent | 2950* |
bosh vm status:
...
| api_z1/0 (03390add-275d-4927-8d5a-8ad3f60729fe) | starting | n/a | large_z1 | 10.90.27.129 |
| api_z2/0 (fc8fb93f-f5be-4389-bdb5-398427f23cef) | failing | n/a | large_z2 | 10.90.28.137 |
...
Looks like cloud_controller_ng, cloud_controller_worker_local_1, cloud_controller_worker_local_2, nginx_cc are failing/initializing.
monit status from api_z1.
Process 'cloud_controller_ng'
status Connection failed
monitoring status monitored
...
Process 'cloud_controller_worker_local_1'
status initializing
monitoring status initializing
data collected Tue Apr 5 23:08:30 2016
Process 'cloud_controller_worker_local_2'
status initializing
monitoring status initializing
data collected Tue Apr 5 23:08:30 2016
Process 'nginx_cc'
status initializing
monitoring status initializing
data collected Tue Apr 5 23:08:30 2016
Logs look similar from both nodes z1/z2.
Logs from /var/vcap/data/sys/log/cloud_controller_ng_ctl.err.log:
[2016-04-05 21:36:18+0000] ------------ STARTING cloud_controller_ng_ctl at Tue Apr 5 21:36:18 UTC 2016 --------------
[2016-04-05 21:36:30+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: already initialized constant AppSubresource::ROLES_FOR_READING
[2016-04-05 21:36:30+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: previous definition of ROLES_FOR_READING was here
[2016-04-05 21:36:31+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: already initialized constant VCAP::CloudController::RestController::BaseController::V2_ROUTE_PREFIX
[2016-04-05 21:36:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: previous definition of V2_ROUTE_PREFIX was here
[2016-04-05 21:36:31+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: already initialized constant VCAP::CloudController::Errors
[2016-04-05 21:36:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: previous definition of Errors was here
[2016-04-05 21:36:31+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: already initialized constant VCAP::CloudController::RestController::ModelController::CENSORED_MESSAGE
[2016-04-05 21:36:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: previous definition of CENSORED_MESSAGE was here
[2016-04-05 21:37:31+0000] ------------ STARTING cloud_controller_ng_ctl at Tue Apr 5 21:37:31 UTC 2016 --------------
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: already initialized constant AppSubresource::ROLES_FOR_READING
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: previous definition of ROLES_FOR_READING was here
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: already initialized constant VCAP::CloudController::RestController::BaseController::V2_ROUTE_PREFIX
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: previous definition of V2_ROUTE_PREFIX was here
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: already initialized constant VCAP::CloudController::Errors
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: previous definition of Errors was here
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: already initialized constant VCAP::CloudController::RestController::ModelController::CENSORED_MESSAGE
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: previous definition of CENSORED_MESSAGE was here
[2016-04-05 21:37:53+0000] ------------ STARTING cloud_controller_ng_ctl at Tue Apr 5 21:37:53 UTC 2016 --------------
Logs from /var/vcap/data/sys/log/cloud_controller_worker_ctl.err.log:
[2016-04-05 21:36:32+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:32 UTC 2016 --------------
[2016-04-05 21:36:33+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:33 UTC 2016 --------------
[2016-04-05 21:36:51+0000] rake aborted!
[2016-04-05 21:36:51+0000] rake aborted!
[2016-04-05 21:36:51+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:36:51+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:36:51+0000] Tasks: TOP => buildpacks:install
[2016-04-05 21:36:51+0000] Tasks: TOP => jobs:local
[2016-04-05 21:36:51+0000] (See full trace by running task with --trace)
[2016-04-05 21:36:51+0000] (See full trace by running task with --trace)
[2016-04-05 21:36:53+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:53 UTC 2016 --------------
[2016-04-05 21:36:54+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:54 UTC 2016 --------------
[2016-04-05 21:37:06+0000] rake aborted!
[2016-04-05 21:37:06+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:37:06+0000] Tasks: TOP => buildpacks:install
[2016-04-05 21:37:06+0000] (See full trace by running task with --trace)
[2016-04-05 21:37:06+0000] rake aborted!
[2016-04-05 21:37:06+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:37:06+0000] Tasks: TOP => jobs:local
[2016-04-05 21:37:06+0000] (See full trace by running task with --trace)
[2016-04-05 21:37:15+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:37:15 UTC 2016 --------------
[2016-04-05 21:37:16+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:37:16 UTC 2016 --------------
[2016-04-05 21:37:29+0000] rake aborted!
[2016-04-05 21:37:29+0000] SignalException: SIGTERM