Hi,

While trying to deploy Cloud Foundry on AWS, I ran into problems related to non-availability of service.

Details:
After "bosh deploy" is successful, I see the following VMs:
+------------------------------------+---------+---------------+---------------+
| Job/index | State | Resource Pool | IPs |
+------------------------------------+---------+---------------+---------------+
| api_z1/0 | running | small_z1 | 10.0.16.4 |
| doppler_z1/0 | running | small_z1 | 10.0.16.6 |
| etcd_z1/0 | running | small_z1 | 10.0.16.104 |
| ha_proxy_z1/0 | running | small_z1 | 10.0.0.11 |
| | | | 52.70.161.122 |
| hm9000_z1/0 | running | small_z1 | 10.0.16.5 |
| loggregator_trafficcontroller_z1/0 | running | small_z1 | 10.0.16.7 |
| nats_z1/0 | running | small_z1 | 10.0.16.103 |
| nfs_z1/0 | running | small_z1 | 10.0.16.105 |
| postgres_z1/0 | running | small_z1 | 10.0.16.101 |
| router_z1/0 | running | small_z1 | 10.0.16.102 |
| runner_z1/0 | running | small_z1 | 10.0.16.9 |
| uaa_z1/0 | running | small_z1 | 10.0.16.8 |
+------------------------------------+---------+---------------+---------------+

when I try to access one of the URL e.g. http://api.cloudXXXX-cfapp.com, I get "503 Service Unavailable".
A "bosh show cf attributes" results in the following:
bosh-cloudfoundry-0.7.6/lib/bosh/cloudfoundry/release_version_cpi.rb:23:in `initialize': CPI aws_cpi not available for version 149 (RuntimeError)

Other cf commands are also failing with similar error:

$ cf target http://api.cloudXXXX-cfapp.com

Setting target to http://api.cloudXXXX-cfapp.com... FAILED
CFoundry::BadResponse: 503: 503 Service Unavailable
No server is available to handle this request.


Something is certainly missing in my config, could someone please help? I confirm that all the VMs listed above are up and running, including the NAT VM.

Thanks much in advance.

"bosh stop" also fails to stop the instance, see error below:


Failed updating job api_z1 > api_z1/0 (canary): Action Failed get_task: Task 79df557d-1357-43c5-4686-b4fa233a6605 result: Stopping Monitored Services: Stopping service route_registrar: Sending stop request to monit: Request failed, response: Response{ StatusCode: 503, Status: '503 Service Unavailable' }

Any help will be highly appreciated.

There are couple of odd things here. Firstly, the command `bosh show cf attributes` is a valid bosh command; when we run it we see the following:

$ bosh show cf attributes
Unknown command: show cf attributes

Secondly, it looks like your cf CLI is in ruby, based the output you posted in the original message:

$ cf target http://api.cloudXXXX-cfapp.com Setting target to http://api.cloudXXXX-cfapp.com... FAILED CFoundry::BadResponse: 503: 503 Service Unavailable No server is available to handle this request.

The cf CLI was rewritten in golang, and the ruby CLI deprecated some years ago.

Can you confirm what versions of bosh and cf CLI you are using locally, as well as the version of cf-release bosh release you are deploying?

bosh version: BOSH 1.3143.0

cf version: cf 5.4.7

I also tried with cf version 6.14.0+2654a47-2015-11-18 but results were same.

cf release:
| Name | Versions | Commit Hash |
+------+----------+-------------+
| cf | 224* | 65621dd0+ |

Can you say what you mean by "results were same"? For cf version 6+, "cf
target http://..." is not even a valid usage of the target command.

All below mentioned cf commands on version 6:

cf api api.1.2.3.4.xip.io
cf api api.myCloudFoundryDomain.com

cf api http://api.1.2.3.4.xip.io
cf api http://api.myCloudFoundryDomain.com

resulted in 503 Service Unavailable.

This is the exact command and error code when running cf v6:

../cf api --skip-ssl-validation api.52.70.161.122.xip.io
Setting api endpoint to api.52.70.161.122.xip.io...
FAILED
Server error, status code: 503, error code: 0, message:

SUCCESS..!!!

I used latest release cf-226.yml and it seemed to work. Now I do not get 503. See below:

../cf api --skip-ssl-validation api.52.70.161.122.xip.io
Setting api endpoint to api.52.70.161.122.xip.io...
OK


API endpoint: https://api.52.70.161.122.xip.io (API version: 2.44.0)
Not logged in. Use 'cf login' to log in.

I tried admin admin and that didnt work. I tried admin CFPASSWORD (this is the password I have configured in my manifest.yml) and didnt work.

"cf login" asks for email and password: is there a default user that I can use to login or do I have to setup UAAC and create a user first?

Find the following in your manifest:

properties:
...
uaa:
...
scim:
...
users:
- admin|ADMIN_PASSWORD|...

You'll find your password there.

I have this in manifest:

uaa:
admin:
client_secret: CFPASSWORD
batch:
password: CFPASSWORD
username: batch_user
cc:
client_secret: CFPASSWORD
scim:
userids_enabled: true
users:
- admin|CFPASSWORD|scim.write,scim.read,openid,cloud_controller.admin,doppler.firehose,routing.router_groups.read

And I tried the following:

../cf login admin
Email> {left blank}
Password> CFPASSWORD

Result = credentials were rejected.

../cf login batch_user
Email> {left blank}
Password> CFPASSWORD

Result = credentials were rejected.

Am I not reading the config in manifest right?

It looks like you're misusing the cf login command.

cf login requires the username to be entered at the 'email' prompt, and the password entered at the 'password prompt.

If you want to script login, you could also use cf auth, which takes the username and password as arguments i.e. cf auth admin CFPASSWORD

Rob & Zak,
cf release integration

Okay, user name at 'email' prompt worked, I wonder if there was a documentation that mentions this.

Also as an fyi, I am not sure what is new in cf-226 but this is the only change that enabled me to successfully deploy CF on EC2.

I have more questions but they are not related to this topic, so this thread can be closed.

Thanks for all your help.

No problem. Glad it's working now. We're not aware of any reason this didn't work with versions before cf v226 - it's probably a conflation of this upgraded version with some other change e.g. a manifest property.

As far as entering `user_name` the the `email prompt` - we agree this is confusing and we're not aware of any documentation that explicitly calls it out. The documentation is actively being worked on, so we will provide this feedback.

Thanks,
Rob Dimsdale
cf release integration

I am sorry that I had to come back to this thread for a recurring issue - the deployment worked fine earlier but every now and then, I keep getting "connection reset by peer" errors when setting api endpoint, see below:

../cf api api.52.20.95.214.xip.io
Setting api endpoint to api.52.20.95.214.xip.io...
FAILED
Error performing request: Get http://api.52.20.95.214.xip.io/v2/info: read tcp 10.212.1.111:42079->52.20.95.214:80: read: connection reset by peer

What could be the reason for this to happen? It occurs 9 out of 10 times when I have to redeploy the whole thing before it succeeds.

Here is the output with CF_TRACE=true:

$ CF_TRACE=true ../cf api api.52.20.95.214.xip.io
Setting api endpoint to api.52.20.95.214.xip.io...

REQUEST: [2015-12-15T06:06:14+02:00]
GET /v2/info HTTP/1.1
Host: api.52.20.95.214.xip.io
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: go-cli 6.14.0+2654a47 / linux



REQUEST: [2015-12-15T06:06:15+02:00]
GET /v2/info HTTP/1.1
Host: api.52.20.95.214.xip.io
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: go-cli 6.14.0+2654a47 / linux


FAILED
Error performing request: Get http://api.52.20.95.214.xip.io/v2/info: read tcp 10.212.1.111:42098->52.20.95.214:80: read: connection reset by peer
FAILED
Error performing request: Get http://api.52.20.95.214.xip.io/v2/info: read tcp 10.212.1.111:42098->52.20.95.214:80: read: connection reset by peer

Hi Rahul,

/info endpoint is failing so the issue can be on cloud controller or
routers or load balancer.

If you are using HAproxy as load balancer then bosh ssh into haproxy and
check if it is failing for some reason , also monitor bosh vms status for
other job role status.

This is just the first step for troubleshooting your issue.

Thanks
Ronak

It looks like HAProxy is terminating the connection from the CLI. This could be for a few reasons - perhaps the HAProxy is misconfigured, or perhaps the backends it is trying to communicate with are unhealthy.

As mentioned in the reply above, getting the HAProxy logs is a good first step. Please also post the output of `bosh vms --vitals` as well as a (sanitized) manifest. If there are any unhealthy VMS you could also get those logs too.

You could also use an AWS Elastic Load Balancer in place of the HAProxy as a way to help determine if the problem is the HAProxy itself or the backends it is trying to route to (e.g. CloudController, router etc). For our own deployments we typically use ELBs instead of HAProxy jobs when deploying to AWS.

Thanks,
Rob & Zak
CF release integration
Pivotal