Bosh deploy failed on AWS-Failed loading settings via fetcher


Mark Wong <mark.wong@...>
 

Trying to deploy Cloud Foundry in AWS. Followed instructions from:
http://bosh.io/docs/deploy-microbosh-to-aws.html
http://docs.cloudfoundry.org/deploying/ec2/configure_aws_cf.html
http://docs.cloudfoundry.org/deploying/ec2/deploy_aws_cf.html

What I found it every VM that BOSH deploy into the public subnet, it failed to response. I end up tried the compilation part of the deploy using VM in public subnet.

The actual deploy of each components failed and I checked the non-responding log in /var/vcap/bosh/log/current and found the following.

2015-06-01_09:01:10.28857 [File System] 2015/06/01 09:01:10 DEBUG - Checking if file exists /sys/class/net/lo/device
2015-06-01_09:01:10.28857 [Cmd Runner] 2015/06/01 09:01:10 DEBUG - Running command: ifup --no-act eth0
2015-06-01_09:01:10.28945 [Cmd Runner] 2015/06/01 09:01:10 DEBUG - Stdout:
2015-06-01_09:01:10.28946 [Cmd Runner] 2015/06/01 09:01:10 DEBUG - Stderr: ifup: interface eth0 already configured
2015-06-01_09:01:10.28946 [Cmd Runner] 2015/06/01 09:01:10 DEBUG - Successful: true (0)
2015-06-01_09:01:12.93269 [settingsService] 2015/06/01 09:01:12 ERROR - Failed loading settings via fetcher: Unmarshalling settings wrapper: invalid character 'I' looking for beginning of value
2015-06-01_09:01:12.93271 [File System] 2015/06/01 09:01:12 DEBUG - Reading file /var/vcap/bosh/settings.json
2015-06-01_09:01:12.93274 [settingsService] 2015/06/01 09:01:12 ERROR - Failed reading settings from file Opening file /var/vcap/bosh/settings.json: open /var/vcap/bosh/settings.json: no such file or directory


I have checked the NAT from private subnet is working and can ping the bosh director host and outside.

Any idea on how to troubleshoot this further?
This email and any attachments to it may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of Citihub Consulting. If you are not the intended recipient of this email, you must neither take any action based upon its contents, nor copy or show it to anyone. Please contact the sender if you believe you have received this email in error.


Dmitriy Kalinin
 

This is not a networking issue but rather some kind of problem with
bootstrapping.

"Failed loading settings via fetcher: Unmarshalling settings wrapper:
invalid character 'I' looking for beginning of value"

It seems that agent fails to retrieve bootstrapping metadata from the
registry:
https://github.com/cloudfoundry/bosh-agent/blob/3dad053a960a677d9fe0e8ee7c58438700a9838a/infrastructure/http_registry.go#L88

Any errors show up in the registry logs on the Director VM?

On Mon, Jun 1, 2015 at 2:27 AM, Mark Wong <mark.wong(a)citihub.com> wrote:

Trying to deploy Cloud Foundry in AWS. Followed instructions from:

http://bosh.io/docs/deploy-microbosh-to-aws.html

http://docs.cloudfoundry.org/deploying/ec2/configure_aws_cf.html

http://docs.cloudfoundry.org/deploying/ec2/deploy_aws_cf.html



What I found it every VM that BOSH deploy into the public subnet, it
failed to response. I end up tried the compilation part of the deploy using
VM in public subnet.



The actual deploy of each components failed and I checked the
non-responding log in /var/vcap/bosh/log/current and found the following.



2015-06-01_09:01:10.28857 [File System] 2015/06/01 09:01:10 DEBUG -
Checking if file exists /sys/class/net/lo/device

2015-06-01_09:01:10.28857 [Cmd Runner] 2015/06/01 09:01:10 DEBUG - Running
command: ifup --no-act eth0

2015-06-01_09:01:10.28945 [Cmd Runner] 2015/06/01 09:01:10 DEBUG - Stdout:

2015-06-01_09:01:10.28946 [Cmd Runner] 2015/06/01 09:01:10 DEBUG - Stderr:
ifup: interface eth0 already configured

2015-06-01_09:01:10.28946 [Cmd Runner] 2015/06/01 09:01:10 DEBUG -
Successful: true (0)

2015-06-01_09:01:12.93269 [settingsService] 2015/06/01 09:01:12 ERROR -
Failed loading settings via fetcher: Unmarshalling settings wrapper:
invalid character 'I' looking for beginning of value

2015-06-01_09:01:12.93271 [File System] 2015/06/01 09:01:12 DEBUG -
Reading file /var/vcap/bosh/settings.json

2015-06-01_09:01:12.93274 [settingsService] 2015/06/01 09:01:12 ERROR -
Failed reading settings from file Opening file
/var/vcap/bosh/settings.json: open /var/vcap/bosh/settings.json: no such
file or directory





I have checked the NAT from private subnet is working and can ping the
bosh director host and outside.



Any idea on how to troubleshoot this further?
This email and any attachments to it may be confidential and are intended
solely for the use of the individual to whom it is addressed. Any views or
opinions expressed are solely those of the author and do not necessarily
represent those of Citihub Consulting. If you are not the intended
recipient of this email, you must neither take any action based upon its
contents, nor copy or show it to anyone. Please contact the sender if you
believe you have received this email in error.

_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh


Mark Wong <mark.wy.wong@...>
 

After I stopped all the VMs include the director nat and the VM running
bosh CLI. I tried redeploy again. VMs get started but soon after I am able
to ssh into them, they get shutdown. I attached the debug.