Hi,
I just deployed v233 in a vSphere environment. Both api_z1 and api_z2 are failing. Any help would be appreciated.

Stemcell version:

| bosh-vsphere-esxi-ubuntu-trusty-go_agent | bosh-vsphere-esxi-ubuntu-trusty-go_agent | 2950* |

bosh vm status:

...
| api_z1/0 (03390add-275d-4927-8d5a-8ad3f60729fe) | starting | n/a | large_z1 | 10.90.27.129 |
| api_z2/0 (fc8fb93f-f5be-4389-bdb5-398427f23cef) | failing | n/a | large_z2 | 10.90.28.137 |
...

Looks like cloud_controller_ng, cloud_controller_worker_local_1, cloud_controller_worker_local_2, nginx_cc are failing/initializing.

monit status from api_z1.

Process 'cloud_controller_ng'
status Connection failed
monitoring status monitored
...
Process 'cloud_controller_worker_local_1'
status initializing
monitoring status initializing
data collected Tue Apr 5 23:08:30 2016

Process 'cloud_controller_worker_local_2'
status initializing
monitoring status initializing
data collected Tue Apr 5 23:08:30 2016

Process 'nginx_cc'
status initializing
monitoring status initializing
data collected Tue Apr 5 23:08:30 2016

Logs look similar from both nodes z1/z2.

Logs from /var/vcap/data/sys/log/cloud_controller_ng_ctl.err.log:

[2016-04-05 21:36:18+0000] ------------ STARTING cloud_controller_ng_ctl at Tue Apr 5 21:36:18 UTC 2016 --------------
[2016-04-05 21:36:30+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: already initialized constant AppSubresource::ROLES_FOR_READING
[2016-04-05 21:36:30+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: previous definition of ROLES_FOR_READING was here
[2016-04-05 21:36:31+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: already initialized constant VCAP::CloudController::RestController::BaseController::V2_ROUTE_PREFIX
[2016-04-05 21:36:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: previous definition of V2_ROUTE_PREFIX was here
[2016-04-05 21:36:31+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: already initialized constant VCAP::CloudController::Errors
[2016-04-05 21:36:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: previous definition of Errors was here
[2016-04-05 21:36:31+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: already initialized constant VCAP::CloudController::RestController::ModelController::CENSORED_MESSAGE
[2016-04-05 21:36:31+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: previous definition of CENSORED_MESSAGE was here
[2016-04-05 21:37:31+0000] ------------ STARTING cloud_controller_ng_ctl at Tue Apr 5 21:37:31 UTC 2016 --------------
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: already initialized constant AppSubresource::ROLES_FOR_READING
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/v3/mixins/app_subresource.rb:5: warning: previous definition of ROLES_FOR_READING was here
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: already initialized constant VCAP::CloudController::RestController::BaseController::V2_ROUTE_PREFIX
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/base_controller.rb:11: warning: previous definition of V2_ROUTE_PREFIX was here
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: already initialized constant VCAP::CloudController::Errors
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/front_controller.rb:7: warning: previous definition of Errors was here
[2016-04-05 21:37:35+0000] /var/vcap/data/packages/cloud_controller_ng/0f8b96021b770958c5e092c2c985a4aeb7977f80.1-489b907bc3437165bc52a390fdc42d37d5c13383/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: already initialized constant VCAP::CloudController::RestController::ModelController::CENSORED_MESSAGE
[2016-04-05 21:37:35+0000] /var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/controllers/base/model_controller.rb:8: warning: previous definition of CENSORED_MESSAGE was here
[2016-04-05 21:37:53+0000] ------------ STARTING cloud_controller_ng_ctl at Tue Apr 5 21:37:53 UTC 2016 --------------

Logs from /var/vcap/data/sys/log/cloud_controller_worker_ctl.err.log:

[2016-04-05 21:36:32+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:32 UTC 2016 --------------
[2016-04-05 21:36:33+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:33 UTC 2016 --------------
[2016-04-05 21:36:51+0000] rake aborted!
[2016-04-05 21:36:51+0000] rake aborted!
[2016-04-05 21:36:51+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:36:51+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:36:51+0000] Tasks: TOP => buildpacks:install
[2016-04-05 21:36:51+0000] Tasks: TOP => jobs:local
[2016-04-05 21:36:51+0000] (See full trace by running task with --trace)
[2016-04-05 21:36:51+0000] (See full trace by running task with --trace)
[2016-04-05 21:36:53+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:53 UTC 2016 --------------
[2016-04-05 21:36:54+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:36:54 UTC 2016 --------------
[2016-04-05 21:37:06+0000] rake aborted!
[2016-04-05 21:37:06+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:37:06+0000] Tasks: TOP => buildpacks:install
[2016-04-05 21:37:06+0000] (See full trace by running task with --trace)
[2016-04-05 21:37:06+0000] rake aborted!
[2016-04-05 21:37:06+0000] Sequel::Migrator::NotCurrentError: migrator is not current
[2016-04-05 21:37:06+0000] Tasks: TOP => jobs:local
[2016-04-05 21:37:06+0000] (See full trace by running task with --trace)
[2016-04-05 21:37:15+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:37:15 UTC 2016 --------------
[2016-04-05 21:37:16+0000] ------------ STARTING cloud_controller_worker_ctl at Tue Apr 5 21:37:16 UTC 2016 --------------
[2016-04-05 21:37:29+0000] rake aborted!
[2016-04-05 21:37:29+0000] SignalException: SIGTERM

Logs from /var/vcap/data/sys/log/nginx_ctl.err.log:

[2016-04-05 21:40:34+0000] ------------ STARTING nginx_ctl at Tue Apr 5 21:40:34 UTC 2016 --------------
[2016-04-05 21:41:30+0000] /var/vcap/packages/capi_utils/utils.sh: line 70: #{timeout}: syntax error: operand expected (error token is #{timeout})
[2016-04-05 21:41:39+0000] ------------ STARTING nginx_ctl at Tue Apr 5 21:41:39 UTC 2016 --------------
[2016-04-05 21:41:52+0000] /var/vcap/packages/capi_utils/utils.sh: line 70: #{timeout}: syntax error: operand expected (error token is #{timeout})

I've tried with version 3215 stemcells - still same issue.

| bosh-vsphere-esxi-ubuntu-trusty-go_agent | ubuntu-trusty | 3215*

Hi Ranga,

Apparently this is related to the following issue: https://github.com/cloudfoundry/cloud_controller_ng/issues/570

I hope this gets fixed in the new release.

Regards,
Omar

Thanks, Omar.

I tried v231. Unfortunately, same issue.

Hi Ranga,

To trouble-shoot this issue could you also provide the contents of
/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log
and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log?
This should give us more details about what's going on. The ctl script logs
do not provide enough details.

Thanks,
Peter

On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

I tried v231. Unfortunately, same issue.

Hi Peter,

Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz
and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a
/var/vcap/sys/log/cloud_controller_worker/ directory on this node.

vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/
agent_ctl.err.log cloud_controller_worker_ctl.log
nginx_cc/
agent_ctl.log consul_agent/
nginx_ctl.err.log
cloud_controller_migration/ metron_agent/
nginx_ctl.log
cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log
route_registrar/
cloud_controller_migration_ctl.log metron_agent_ctl.log
route_registrar_ctl.err.log
cloud_controller_ng/ monit/
route_registrar_ctl.log
cloud_controller_ng_ctl.err.log nfs_mounter/
statsd-injector/
cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log
statsd-injector-ctl.err.log
cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log
statsd-injector-ctl.log

--
Thanks,

Ranga

Hi Ranga,

Looking at your logs we found an error that could possibly cause this and
it is related to the properties.apps_domain in the deployment manifest. By
setting it to 'b%()osh-lite.com' (using special characters), we could
reproduce the following error which we also found in your log file:

Encountered error: name
format\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/model/base.rb:1543:in
`save'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:35:in
`block in
find_or_create'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:134:in
`_transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:108:in
`block in
transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in
`block in
synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/connection_pool/threaded.rb:103:in
`hold'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in
`synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:97:in
`transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:27:in
`find_or_create'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:57:in
`block in
create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in
`each'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in
`create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:9:in
`write_seed_data'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:93:in
`block in
run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`call'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`run_machine'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`run'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:87:in
`run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:8:in
`<main>'

Can you check the apps_domain property and see if there is anything
suspicious with it?

Thanks,
Peter & Kara

On Thu, Apr 7, 2016 at 2:32 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

Hi Peter,

Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz
and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a
/var/vcap/sys/log/cloud_controller_worker/ directory on this node.

vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/
agent_ctl.err.log cloud_controller_worker_ctl.log
nginx_cc/
agent_ctl.log consul_agent/
nginx_ctl.err.log
cloud_controller_migration/ metron_agent/
nginx_ctl.log
cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log
route_registrar/
cloud_controller_migration_ctl.log metron_agent_ctl.log
route_registrar_ctl.err.log
cloud_controller_ng/ monit/
route_registrar_ctl.log
cloud_controller_ng_ctl.err.log nfs_mounter/
statsd-injector/
cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log
statsd-injector-ctl.err.log
cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log
statsd-injector-ctl.log



On Thu, Apr 7, 2016 at 12:21 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote:

Hi Ranga,

To trouble-shoot this issue could you also provide the contents of
/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log
and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log?
This should give us more details about what's going on. The ctl script logs
do not provide enough details.

Thanks,
Peter

On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

I tried v231. Unfortunately, same issue.

--
Thanks,

Ranga

Here's my deployment manifest. app_domains is set to cfapp. I can't find
bost-lite anywhere in the file at all.

--
Thanks,

Ranga

Hi Ranga,

The only reason we were using bosh-lite.com for our app_domains was because
we were testing to reproduce on our local bosh-lite.
Using 'cfapp' I managed to reproduce the issue locally. My guess is that
this is because 'cfapp' is not a valid domain (it doesn't end with a valid
top-level domain), and I'm guessing that fixing that will at least fix part
of the problem if not all of it.

Thanks,

Kara

On Thu, Apr 7, 2016 at 8:27 PM, Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

Here's my deployment manifest. app_domains is set to cfapp. I can't find
bost-lite anywhere in the file at all.

On Thu, Apr 7, 2016 at 4:27 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote:

Hi Ranga,

Looking at your logs we found an error that could possibly cause this and
it is related to the properties.apps_domain in the deployment manifest. By
setting it to 'b%()osh-lite.com' (using special characters), we could
reproduce the following error which we also found in your log file:

Encountered error: name
format\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/model/base.rb:1543:in
`save'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:35:in
`block in
find_or_create'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:134:in
`_transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:108:in
`block in
transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in
`block in
synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/connection_pool/threaded.rb:103:in
`hold'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in
`synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:97:in
`transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:27:in
`find_or_create'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:57:in
`block in
create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in
`each'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in
`create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:9:in
`write_seed_data'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:93:in
`block in
run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`call'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`run_machine'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`run'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:87:in
`run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:8:in
`<main>'

Can you check the apps_domain property and see if there is anything
suspicious with it?

Thanks,
Peter & Kara

On Thu, Apr 7, 2016 at 2:32 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

Hi Peter,

Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz
and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a
/var/vcap/sys/log/cloud_controller_worker/ directory on this node.

vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/
agent_ctl.err.log cloud_controller_worker_ctl.log
nginx_cc/
agent_ctl.log consul_agent/
nginx_ctl.err.log
cloud_controller_migration/ metron_agent/
nginx_ctl.log
cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log
route_registrar/
cloud_controller_migration_ctl.log metron_agent_ctl.log
route_registrar_ctl.err.log
cloud_controller_ng/ monit/
route_registrar_ctl.log
cloud_controller_ng_ctl.err.log nfs_mounter/
statsd-injector/
cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log
statsd-injector-ctl.err.log
cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log
statsd-injector-ctl.log



On Thu, Apr 7, 2016 at 12:21 PM, Peter Goetz <peter.gtz(a)gmail.com>
wrote:

Hi Ranga,

To trouble-shoot this issue could you also provide the contents of
/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log
and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log?
This should give us more details about what's going on. The ctl script logs
do not provide enough details.

Thanks,
Peter

On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

I tried v231. Unfortunately, same issue.

--
Thanks,

Ranga

--
Thanks,

Ranga

HI Kara,

Thanks. Let me try with a valid app_domain.

On Thu, Apr 7, 2016 at 9:01 PM, Kara Alexandra <ardnaxelarak(a)gmail.com>
wrote:

Hi Ranga,

The only reason we were using bosh-lite.com for our app_domains was
because we were testing to reproduce on our local bosh-lite.
Using 'cfapp' I managed to reproduce the issue locally. My guess is that
this is because 'cfapp' is not a valid domain (it doesn't end with a valid
top-level domain), and I'm guessing that fixing that will at least fix part
of the problem if not all of it.

Thanks,

Kara

On Thu, Apr 7, 2016 at 8:27 PM, Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

Here's my deployment manifest. app_domains is set to cfapp. I can't find
bost-lite anywhere in the file at all.

On Thu, Apr 7, 2016 at 4:27 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote:

Hi Ranga,

Looking at your logs we found an error that could possibly cause this
and it is related to the properties.apps_domain in the deployment manifest.
By setting it to 'b%()osh-lite.com' (using special characters), we
could reproduce the following error which we also found in your log file:

Encountered error: name
format\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/model/base.rb:1543:in
`save'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:35:in
`block in
find_or_create'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:134:in
`_transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:108:in
`block in
transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in
`block in
synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/connection_pool/threaded.rb:103:in
`hold'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in
`synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:97:in
`transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:27:in
`find_or_create'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:57:in
`block in
create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in
`each'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in
`create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:9:in
`write_seed_data'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:93:in
`block in
run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`call'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`run_machine'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`run'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:87:in
`run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:8:in
`<main>'

Can you check the apps_domain property and see if there is anything
suspicious with it?

Thanks,
Peter & Kara

On Thu, Apr 7, 2016 at 2:32 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

Hi Peter,

Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz
and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a
/var/vcap/sys/log/cloud_controller_worker/ directory on this node.

vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/
agent_ctl.err.log cloud_controller_worker_ctl.log
nginx_cc/
agent_ctl.log consul_agent/
nginx_ctl.err.log
cloud_controller_migration/ metron_agent/
nginx_ctl.log
cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log
route_registrar/
cloud_controller_migration_ctl.log metron_agent_ctl.log
route_registrar_ctl.err.log
cloud_controller_ng/ monit/
route_registrar_ctl.log
cloud_controller_ng_ctl.err.log nfs_mounter/
statsd-injector/
cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log
statsd-injector-ctl.err.log
cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log
statsd-injector-ctl.log



On Thu, Apr 7, 2016 at 12:21 PM, Peter Goetz <peter.gtz(a)gmail.com>
wrote:

Hi Ranga,

To trouble-shoot this issue could you also provide the contents of
/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log
and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log?
This should give us more details about what's going on. The ctl script logs
do not provide enough details.

Thanks,
Peter

On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

I tried v231. Unfortunately, same issue.

--
Thanks,

Ranga

--
Thanks,

Ranga

--
Thanks,

Ranga

Hi Kara & Peter,

Thanks a lot for your help. That fixed the issue.

On Thu, Apr 7, 2016 at 9:04 PM, Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

HI Kara,

Thanks. Let me try with a valid app_domain.

On Thu, Apr 7, 2016 at 9:01 PM, Kara Alexandra <ardnaxelarak(a)gmail.com>
wrote:

Hi Ranga,

The only reason we were using bosh-lite.com for our app_domains was
because we were testing to reproduce on our local bosh-lite.
Using 'cfapp' I managed to reproduce the issue locally. My guess is that
this is because 'cfapp' is not a valid domain (it doesn't end with a valid
top-level domain), and I'm guessing that fixing that will at least fix part
of the problem if not all of it.

Thanks,

Kara

On Thu, Apr 7, 2016 at 8:27 PM, Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

Here's my deployment manifest. app_domains is set to cfapp. I can't find
bost-lite anywhere in the file at all.

On Thu, Apr 7, 2016 at 4:27 PM, Peter Goetz <peter.gtz(a)gmail.com> wrote:

Hi Ranga,

Looking at your logs we found an error that could possibly cause this
and it is related to the properties.apps_domain in the deployment manifest.
By setting it to 'b%()osh-lite.com' (using special characters), we
could reproduce the following error which we also found in your log file:

Encountered error: name
format\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/model/base.rb:1543:in
`save'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:35:in
`block in
find_or_create'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:134:in
`_transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:108:in
`block in
transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in
`block in
synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/connection_pool/threaded.rb:103:in
`hold'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/connecting.rb:249:in
`synchronize'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/sequel-4.29.0/lib/sequel/database/transactions.rb:97:in
`transaction'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/app/models/runtime/shared_domain.rb:27:in
`find_or_create'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:57:in
`block in
create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in
`each'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:56:in
`create_seed_domains'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/seeds.rb:9:in
`write_seed_data'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:93:in
`block in
run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`call'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`run_machine'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.2.0/gems/eventmachine-1.0.9.1/lib/eventmachine.rb:193:in
`run'\n/var/vcap/data/packages/cloud_controller_ng/da452b34d79be56a0784c7e88d6b9c0e1811a9d8.1-f934c136e6019cf54ab7aa04a3c153657226e729/cloud_controller_ng/lib/cloud_controller/runner.rb:87:in
`run!'\n/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/bin/cloud_controller:8:in
`<main>'

Can you check the apps_domain property and see if there is anything
suspicious with it?

Thanks,
Peter & Kara

On Thu, Apr 7, 2016 at 2:32 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

Hi Peter,

Attaching /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log.gz
and /var/vcap/sys/log/cloud_controller_worker_ctl.log.gz. There isn't a
/var/vcap/sys/log/cloud_controller_worker/ directory on this node.

vcap(a)572afc33-0735-4727-8ff3-9dc6d7fa8af0:~$ ls /var/vcap/sys/log/
agent_ctl.err.log
cloud_controller_worker_ctl.log nginx_cc/
agent_ctl.log consul_agent/
nginx_ctl.err.log
cloud_controller_migration/ metron_agent/
nginx_ctl.log
cloud_controller_migration_ctl.err.log metron_agent_ctl.err.log
route_registrar/
cloud_controller_migration_ctl.log metron_agent_ctl.log
route_registrar_ctl.err.log
cloud_controller_ng/ monit/
route_registrar_ctl.log
cloud_controller_ng_ctl.err.log nfs_mounter/
statsd-injector/
cloud_controller_ng_ctl.log nfs_mounter_ctl.err.log
statsd-injector-ctl.err.log
cloud_controller_worker_ctl.err.log nfs_mounter_ctl.log
statsd-injector-ctl.log



On Thu, Apr 7, 2016 at 12:21 PM, Peter Goetz <peter.gtz(a)gmail.com>
wrote:

Hi Ranga,

To trouble-shoot this issue could you also provide the contents of
/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log
and /var/vcap/sys/log/cloud_controller_worker/cloud_controller_worker.log?
This should give us more details about what's going on. The ctl script logs
do not provide enough details.

Thanks,
Peter

On Wed, Apr 6, 2016 at 6:12 PM Ranga Rajagopalan <
ranga.rajagopalan(a)gmail.com> wrote:

I tried v231. Unfortunately, same issue.

--
Thanks,

Ranga

--
Thanks,

Ranga

--
Thanks,

Ranga

--
Thanks,

Ranga