Are you using centos ? /etc/resolvconf/resolv.conf.d/head On Wed, Jun 17, 2015 at 2:21 PM, Ahmed Ali (ahmeali) <ahmeali(a)cisco.com> wrote: These are the errors I found under /var/vcap/sys/log:
/var/vcap/sys/log/consul_template_ctl.err.log:
[2015-06-17 05:10:03+0000] ------------ STARTING consul_template_ctl at
Wed Jun 17 05:10:03 UTC 2015 --------------
[2015-06-17 05:10:03+0000] 2015/06/17 05:10:03 [ERR] (runner) watcher
reported error: health services: error fetching: Get
http://127.0.0.1:8500/v1/health/service/ssh-proxy?wait=60000ms: dial tcp
127.0.0.1:8500: connection refused
[2015-06-17 05:10:08+0000] 2015/06/17 05:10:08 [ERR] (runner) watcher
reported error: health services: error fetching: Get
http://127.0.0.1:8500/v1/health/service/ssh-proxy?wait=60000ms: dial tcp
127.0.0.1:8500: connection refused
[2015-06-17 05:10:13+0000] 2015/06/17 05:10:13 [ERR] (runner) watcher
reported error: health services: error fetching: Get
http://127.0.0.1:8500/v1/health/service/ssh-proxy?wait=60000ms: dial tcp
127.0.0.1:8500: connection refused
[2015-06-17 05:10:14+0000] Received interrupt, cleaning up...
[2015-06-17 05:10:14+0000] /var/vcap/jobs/haproxy/bin/consul_template_ctl:
line 46: exit: : numeric argument required
/var/vcap/sys/log/monit/consul_agent.err.log:
/var/vcap/jobs/consul_agent/bin/agent_ctl: line 39:
/etc/resolvconf/resolv.conf.d/head: No such file or directory
/var/vcap/sys/log/metron_agent/metron_agent.stdout.log:
{"timestamp":1434518129.113139153,"process_id":2263,"source":"metron","log_level":"warn","message":"Failed
to create client: Could not connect to NATS: nats: No servers available for
connection","data":null,"file":"/var/vcap/data/compile/metron_agent/loggregator/src/
github.com/cloudfoundry/loggregatorlib/cfcomponent/registrars/collectorregistrar/collector_registrar.go
","line":51,"method":"
github.com/cloudfoundry/loggregatorlib/cfcomponent/registrars/collectorregistrar.(*CollectorRegistrar).Run
”}
Note: There is a proxy server between all CF VMS and the internet, is
there any connections to the outside? If so where in cf-stub.yml proxy can
be placed.
Thank you!
From: Gwenn Etourneau <getourneau(a)pivotal.io>
Reply-To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Date: Tuesday, June 16, 2015 at 6:56 PM
To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Subject: Re: [cf-bosh] cf-stub.yml example with minimum or required info
bosh ssh to the haproxy and check the logs /var/vcap/sys/log.
On Wed, Jun 17, 2015 at 5:14 AM, Ahmed Ali (ahmeali) <ahmeali(a)cisco.com>
wrote:
Thank you!
After adding environment to meta section, the deployment is moving
forward but timing out on “Started updating job ha_proxy_z1 > ha_proxy_z1/0
(canary)"
What changed:
meta:
environment: cf
properties:
domain: foundry-appx.company.com #(domain used in the ssl cert)
I think this could be related to SSL cert and keys which should be
included in cf-stub.yml, I created SSL certificate for “ssl_pem” by
following this link
<https://github.com/cloudfoundry/cf-release/tree/master/example_manifests> ,
also added the jwt signing_key which was created using “ssh-keygen -t rsa”,
I see another place (here
<http://docs.cloudfoundry.org/deploying/cf-stub-vsphere.html>) where SSL
cert/key are needed in cf-stub.yml under “router > ssl_cert:” but not
sure if it is the same as ssl_pem, any idea what Im missing here?
"bosh cck" is coming out clean and no problems, "bosh vms" show all
VMs in running state except “ha_proxy_z1” in failing state.
# bosh deploy
Processing deployment manifest
------------------------------
Getting deployment properties from director...
Compiling deployment manifest...
Please review all changes carefully
Deploying
---------
Deployment name: `cf-deployment.yml'
Director name: `bosh2'
Are you sure you want to deploy? (type 'yes' to continue): yes
Director task 227
Started unknown
Started unknown > Binding deployment. Done (00:00:00)
Started preparing deployment
Started preparing deployment > Binding releases. Done (00:00:01)
Started preparing deployment > Binding existing deployment. Done
(00:00:00)
Started preparing deployment > Binding resource pools. Done (00:00:00)
Started preparing deployment > Binding stemcells. Done (00:00:00)
Started preparing deployment > Binding templates. Done (00:00:00)
Started preparing deployment > Binding properties. Done (00:00:00)
Started preparing deployment > Binding unallocated VMs. Done (00:00:00)
Started preparing deployment > Binding instance networks. Done
(00:00:00)
Started preparing package compilation > Finding packages to compile.
Done (00:00:00)
Started preparing dns > Binding DNS. Done (00:00:00)
Started preparing configuration > Binding configuration. Done
(00:00:02)
Started updating job ha_proxy_z1 > ha_proxy_z1/0 (canary). Failed: `ha_proxy_z1/0'
is not running after update (00:10:18)
Error 400007: `ha_proxy_z1/0' is not running after update
bosh task 227 --debug
E, [2015-06-16 19:41:51 #13416] [canary_update(ha_proxy_z1/0)] *ERROR*
-- DirectorJobRunner: *Error* updating canary instance:
#<Bosh::Director::AgentJobNotRunning: `ha_proxy_z1/0' is not running after
update>
I, [2015-06-16 19:41:51 #13416] [task:227] INFO -- DirectorJobRunner:
sending update deployment *error* event
D, [2015-06-16 19:41:51 #13416] [task:227] DEBUG -- DirectorJobRunner:
SENT: hm.director.alert
{"id":"52682ea9-0fa2-4edb-9611-128490279ba5","severity":3,"title":"director
- *error* during update deployment","summary":"*Error* during update
deployment for 'cf' against Director
'b9a1bf7b-952f-48e1-a496-f6543d7a782c':
#<Bosh::Director::AgentJobNotRunning: `ha_proxy_z1/0' is not running after
update>","created_at":1434483711}
E, [2015-06-16 19:41:51 #13416] [task:227] *ERROR* -- DirectorJobRunner:
`ha_proxy_z1/0' is not running after update
D, [2015-06-16 19:41:51 #13416] [task:227] DEBUG -- DirectorJobRunner:
(0.000495s) UPDATE "tasks" SET "state" = '*error*', "timestamp" =
'2015-06-16 19:41:51.761481+0000', "description" = 'create deployment',
"result" = '`ha_proxy_z1/0'' is not running after update', "output" =
'/var/vcap/store/director/tasks/227', "checkpoint_time" = '2015-06-16
19:41:29.218401+0000', "type" = 'update_deployment', "username" = 'admin'
WHERE ("id" = 227)
Task 227 *error*
Thanks!
From: CF Runtime <cfruntime(a)gmail.com>
Reply-To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Date: Monday, June 15, 2015 at 2:10 PM
To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Subject: Re: [cf-bosh] cf-stub.yml example with minimum or required info
Hi Ahmed,
This property is coming from the "templates/cf-lamb.yml" within
cf-release. You are able to overwrite this property in your stub like so:
meta:
environment: [name-of-environment]
Hope this helps,
Dan && James, CF Runtime Team
On Sun, Jun 14, 2015 at 7:13 PM, Ahmed Ali (ahmeali) <ahmeali(a)cisco.com>
wrote:
What should be the value then, it is generated by spiff?
should I place this in cf-stub.yml under properties to overwrite what
spiff doing as following:
properties:
metron_agent:
deployment: <???>
I found this link talking about same issue:
https://github.com/cloudfoundry/bosh-lite/issues/265 but could not find
an answer
Thanks Gwenn
From: Gwenn Etourneau <getourneau(a)pivotal.io>
Reply-To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Date: Sunday, June 14, 2015 at 6:58 PM
To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Subject: Re: [cf-bosh] cf-stub.yml example with minimum or required info
Seems this could not be null in deployment.yml
metron_agent:
deployment: null
On Mon, Jun 15, 2015 at 10:50 AM, Ahmed Ali (ahmeali) <ahmeali(a)cisco.com
wrote: Both cf-stub.yml and the spiff generated cf-deployment.yml are
attached.
I noticed in cf-deployment.yml a section called “- default_networks:”
inserted between jobs, does this look normal?
Note: I did not edit cf-deployment, it is what I get from spiff.
Environment info:
Ubuntu 14
BOSH 1.2977.0
cf version 6.11.3-cebadc9-2015-05-20T19:00:58+00:00
spiff version 1.0.6
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]
vSphere 5.5
#bosh status
Config
/root/.bosh_config
Director
Name bosh2
URL https://10.195.166.12:25555
Version 1.2976.0 (00000000)
User admin
UUID b9a1bf7b-952f-48e1-a496-f6543d7a782c
CPI vsphere
dns enabled (domain_name: bosh)
compiled_package_cache disabled
snapshots enabled
Deployment
Manifest /root/deployment/cf-deployment.yml
#bosh releases
+------+------------+-------------+
| Name | Versions | Commit Hash |
+------+------------+-------------+
| cf | 211+dev.1* | 2121dc64+ |
+------+------------+-------------+
(*) Currently deployed
(+) Uncommitted changes
Thank you!
From: Gwenn Etourneau <getourneau(a)pivotal.io>
Reply-To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Date: Sunday, June 14, 2015 at 5:35 PM
To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Subject: Re: [cf-bosh] cf-stub.yml example with minimum or required
info
Please show us you manifest, seems something is missing.
On Mon, Jun 15, 2015 at 4:08 AM, Ahmed Ali (ahmeali) <ahmeali(a)cisco.com
wrote: Hi Joseph,
Thank you! I changed to using two different networks and now "bosh
deploy” works and all VMs are deployed successfully, but looks like there
is a binding configuration error:
Binding configuration. Failed: Error filling in template
`metron_agent.json.erb' for `ha_proxy_z1/0' (line 5: Can't find property
`["metron_agent.deployment"]') (00:00:00)
Error 100: Error filling in template `metron_agent.json.erb' for
`ha_proxy_z1/0' (line 5: Can't find property `["metron_agent.deployment"]')
Also tried to connect using CF and could not connect:
root(a)cloudfoundry:~/deployment# cf api --skip-ssl-validation
foundry-appx.domain.com
Setting api endpoint to *foundry-app.domain.com
<http://foundry-app.domain.com>*...
*FAILED*
Error performing request: Get http://foundry-appx.domain.com/v2/info:
dial tcp x.x.166.18:80: connection refused
Full run console log:
root(a)cloudfoundry:~/deployment# bosh deploy
Processing deployment manifest
------------------------------
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Compiling deployment manifest...
Cannot get current deployment information from director, possibly a
new deployment
Please review all changes carefully
Deploying
---------
Deployment name: `cf-deployment.yml'
Director name: `bosh2'
Are you sure you want to deploy? (type 'yes' to continue): yes
Director task 172
Started unknown
Started unknown > Binding deployment. Done (00:00:00)
Started preparing deployment
Started preparing deployment > Binding releases. Done (00:00:00)
Started preparing deployment > Binding existing deployment. Done
(00:00:00)
Started preparing deployment > Binding resource pools. Done
(00:00:00)
Started preparing deployment > Binding stemcells. Done (00:00:00)
Started preparing deployment > Binding templates. Done (00:00:00)
Started preparing deployment > Binding properties. Done (00:00:00)
Started preparing deployment > Binding unallocated VMs. Done
(00:00:00)
Started preparing deployment > Binding instance networks. Done
(00:00:00)
Started preparing package compilation > Finding packages to compile.
Done (00:00:00)
Started preparing dns > Binding DNS. Done (00:00:00)
Started creating bound missing vms
Started creating bound missing vms > small_z1/0
Started creating bound missing vms > small_z1/1
Started creating bound missing vms > small_z1/2
Started creating bound missing vms > small_z2/0
Started creating bound missing vms > small_z2/1
Started creating bound missing vms > medium_z1/0
Started creating bound missing vms > medium_z1/1
Started creating bound missing vms > medium_z1/2
Started creating bound missing vms > medium_z1/3
Started creating bound missing vms > medium_z1/4
Started creating bound missing vms > medium_z1/5
Started creating bound missing vms > medium_z1/6
Started creating bound missing vms > medium_z1/7
Started creating bound missing vms > medium_z1/8
Started creating bound missing vms > medium_z2/0
Started creating bound missing vms > medium_z2/1
Started creating bound missing vms > medium_z2/2
Started creating bound missing vms > medium_z2/3
Started creating bound missing vms > medium_z2/4
Started creating bound missing vms > large_z1/0
Started creating bound missing vms > large_z2/0
Started creating bound missing vms > runner_z1/0
Started creating bound missing vms > runner_z2/0
Started creating bound missing vms > router_z1/0
Started creating bound missing vms > router_z1/1
Started creating bound missing vms > router_z2/0
Done creating bound missing vms > medium_z1/0 (00:00:30)
Done creating bound missing vms > medium_z1/5 (00:00:32)
Done creating bound missing vms > small_z2/1 (00:00:34)
Done creating bound missing vms > medium_z1/2 (00:00:34)
Done creating bound missing vms > medium_z2/4 (00:00:35)
Done creating bound missing vms > medium_z1/1 (00:00:45)
Done creating bound missing vms > medium_z1/7 (00:00:45)
Done creating bound missing vms > small_z1/0 (00:00:46)
Done creating bound missing vms > router_z1/1 (00:00:47)
Done creating bound missing vms > medium_z2/2 (00:00:49)
Done creating bound missing vms > medium_z2/3 (00:00:49)
Done creating bound missing vms > large_z2/0 (00:00:51)
Done creating bound missing vms > medium_z1/4 (00:00:52)
Done creating bound missing vms > router_z1/0 (00:00:52)
Done creating bound missing vms > small_z1/1 (00:00:55)
Done creating bound missing vms > router_z2/0 (00:00:55)
Done creating bound missing vms > small_z2/0 (00:00:59)
Done creating bound missing vms > large_z1/0 (00:00:59)
Done creating bound missing vms > medium_z2/1 (00:01:00)
Done creating bound missing vms > medium_z1/6 (00:01:00)
Done creating bound missing vms > medium_z1/3 (00:01:01)
Done creating bound missing vms > medium_z1/8 (00:01:01)
Done creating bound missing vms > medium_z2/0 (00:01:01)
Done creating bound missing vms > runner_z2/0 (00:01:02)
Done creating bound missing vms > runner_z1/0 (00:01:02)
Done creating bound missing vms > small_z1/2 (00:01:03)
Done creating bound missing vms (00:01:03)
Started binding instance vms
Started binding instance vms > ha_proxy_z1/0
Started binding instance vms > nats_z1/0
Started binding instance vms > nats_z2/0
Started binding instance vms > etcd_z1/0
Started binding instance vms > etcd_z1/1
Started binding instance vms > etcd_z2/0
Started binding instance vms > stats_z1/0
Started binding instance vms > nfs_z1/0
Started binding instance vms > postgres_z1/0
Started binding instance vms > uaa_z1/0
Started binding instance vms > uaa_z2/0
Started binding instance vms > api_z1/0
Started binding instance vms > api_z2/0
Started binding instance vms > clock_global/0
Started binding instance vms > api_worker_z1/0
Started binding instance vms > api_worker_z2/0
Started binding instance vms > hm9000_z1/0
Started binding instance vms > hm9000_z2/0
Started binding instance vms > runner_z1/0
Started binding instance vms > runner_z2/0
Started binding instance vms > loggregator_z1/0
Started binding instance vms > loggregator_z2/0
Started binding instance vms > loggregator_trafficcontroller_z1/0
Started binding instance vms > router_z1/0
Started binding instance vms > loggregator_trafficcontroller_z2/0
Started binding instance vms > router_z2/0
Done binding instance vms > etcd_z2/0 (00:00:00)
Done binding instance vms > ha_proxy_z1/0 (00:00:00)
Done binding instance vms > nats_z1/0 (00:00:00)
Done binding instance vms > nats_z2/0 (00:00:00)
Done binding instance vms > clock_global/0 (00:00:00)
Done binding instance vms > etcd_z1/0 (00:00:00)
Done binding instance vms > uaa_z1/0 (00:00:00)
Done binding instance vms > nfs_z1/0 (00:00:00)
Done binding instance vms > postgres_z1/0 (00:00:00)
Done binding instance vms > api_z2/0 (00:00:00)
Done binding instance vms > api_z1/0 (00:00:00)
Done binding instance vms > uaa_z2/0 (00:00:00)
Done binding instance vms > etcd_z1/1 (00:00:00)
Done binding instance vms > stats_z1/0 (00:00:00)
Done binding instance vms > hm9000_z2/0 (00:00:00)
Done binding instance vms > hm9000_z1/0 (00:00:00)
Done binding instance vms > runner_z1/0 (00:00:00)
Done binding instance vms > loggregator_z1/0 (00:00:00)
Done binding instance vms > loggregator_z2/0 (00:00:00)
Done binding instance vms > loggregator_trafficcontroller_z1/0
(00:00:00)
Done binding instance vms > loggregator_trafficcontroller_z2/0
(00:00:00)
Done binding instance vms > runner_z2/0 (00:00:00)
Done binding instance vms > router_z2/0 (00:00:00)
Done binding instance vms > router_z1/0 (00:00:00)
Done binding instance vms > api_worker_z1/0 (00:00:01)
Done binding instance vms > api_worker_z2/0 (00:00:01)
Done binding instance vms (00:00:01)
Started preparing configuration > Binding configuration. Failed: Error
filling in template `metron_agent.json.erb' for `ha_proxy_z1/0' (line 5:
Can't find property `["metron_agent.deployment"]') (00:00:00)
Error 100: Error filling in template `metron_agent.json.erb' for
`ha_proxy_z1/0' (line 5: Can't find property `["metron_agent.deployment"]')
Task 172 error
For a more detailed error report, run: bosh task 172 --debug
root(a)cloudfoundry:~/deployment# bosh cck
Performing cloud check...
Processing deployment manifest
------------------------------
Director task 173
Started scanning 26 vms
Started scanning 26 vms > Checking VM states. Done (00:00:00)
Started scanning 26 vms > 26 OK, 0 unresponsive, 0 missing, 0
unbound, 0 out of sync. Done (00:00:00)
Done scanning 26 vms (00:00:00)
Started scanning 0 persistent disks
Started scanning 0 persistent disks > Looking for inactive disks.
Done (00:00:00)
Started scanning 0 persistent disks > 0 OK, 0 missing, 0 inactive,
0 mount-info mismatch. Done (00:00:00)
Done scanning 0 persistent disks (00:00:00)
Task 173 done
Started2015-06-14 18:34:55 UTC
Finished2015-06-14 18:34:55 UTC
Duration00:00:00
Scan is complete, checking if any problems found...
No problems found
root(a)cloudfoundry:~/deployment# cf api --skip-ssl-validation
foundry-appx.domain.com
Setting api endpoint to *foundry-app.doamin.com
<http://foundry-app.doamin.com>*...
*FAILED*
Error performing request: Get http://foundry-appx.domain.com/v2/info:
dial tcp x.x.166.18:80: connection refused
Note: the cluster is done inside a lab environment with proxy, did
not configure CF with proxy and not sure if I need to do something specific.
Thanks!
From: CF Runtime <cfruntime(a)gmail.com>
Reply-To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Date: Friday, June 12, 2015 at 5:39 PM
To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Subject: Re: [cf-bosh] cf-stub.yml example with minimum or required
info
Anything in the network section that is not static or reserved,
BOSH will assume it can use for any other instances in that zone. Because
your two subnets overlap, and you have not partitioned it off using the
reserved sections, BOSH is using that IP for something in the other zone.
Normally, if you only have a single network, it is easier to just
set the instance count for jobs in the second zone to zero, and scale up
any in the first zone to multiple if you want to have redundancy.
Joseph Palermo
CF Runtime Team
On Fri, Jun 12, 2015 at 1:29 PM, Ahmed Ali (ahmeali) <
ahmeali(a)cisco.com> wrote:
Sorry I did not see this reply from Gwenn Etourneau.
After “bosh cck”, it found 6 problems and then I tried option 2 to
reboot and also tried option 3 to recreate VM, none of them fixed the
issue, but I noticed the problematic VMs are using duplicate IPs, my
network section in the manifest has 2 networks (cf1 and cf2) and there is
no overlab, see network section below.
For example the VM router_z1/0
(vm-c36efd49-7ac7-4b90-9779-b5192408e4a6) got the ip
10.195.166.110, and another VM from same deployment (
vm-2b532db0-36df-433d-8132-89c76a9c81c3) got the same ip
10.195.166.110
I tried also removing the section “jobs" and go with defaults which
is generated by spiff and run into same issue, do I have
to statically assign IP address to each job?
M-20JW:cf-release ahmeali$ bosh cck
Performing cloud check...
Processing deployment manifest
------------------------------
Director task 141
Started scanning 26 vms
Started scanning 26 vms > Checking VM states. Done (00:00:10)
Started scanning 26 vms > 20 OK, 6 unresponsive, 0 missing, 0
unbound, 0 out of sync. Done (00:00:00)
Done scanning 26 vms (00:00:10)
Started scanning 0 persistent disks
Started scanning 0 persistent disks > Looking for inactive disks.
Done (00:00:00)
Started scanning 0 persistent disks > 0 OK, 0 missing, 0 inactive,
0 mount-info mismatch. Done (00:00:00)
Done scanning 0 persistent disks (00:00:00)
Task 141 done
Started 2015-06-12 20:03:17 UTC
Finished 2015-06-12 20:03:27 UTC
Duration 00:00:10
Scan is complete, checking if any problems found...
Found 6 problems
Problem 1 of 6: Unknown VM
(vm-e4da0933-52ba-473f-903d-a9ee09d1671f) is not responding.
1. Ignore problem
2. Reboot VM
3. Recreate VM using last known apply spec
4. Delete VM reference (DANGEROUS!)
Please choose a resolution [1 - 4]: 2
Problem 2 of 6: hm9000_z1/0
(vm-5ea4c90d-247e-43ad-a189-ff4d2d781854) is not responding.
1. Ignore problem
2. Reboot VM
3. Recreate VM using last known apply spec
4. Delete VM reference (DANGEROUS!)
Please choose a resolution [1 - 4]: 2
Problem 3 of 6: router_z1/0
(vm-c36efd49-7ac7-4b90-9779-b5192408e4a6) is not responding.
1. Ignore problem
2. Reboot VM
3. Recreate VM using last known apply spec
4. Delete VM reference (DANGEROUS!)
Please choose a resolution [1 - 4]: 2
Problem 4 of 6: nfs_z1/0 (vm-868e2bb9-ac61-49f0-86fb-38d5c338201b)
is not responding.
1. Ignore problem
2. Reboot VM
3. Recreate VM using last known apply spec
4. Delete VM reference (DANGEROUS!)
Please choose a resolution [1 - 4]: 2
Problem 5 of 6: router_z2/0
(vm-773e83ee-c97f-4aa4-b163-d09a703a4678) is not responding.
1. Ignore problem
2. Reboot VM
3. Recreate VM using last known apply spec
4. Delete VM reference (DANGEROUS!)
Please choose a resolution [1 - 4]: 2
Problem 6 of 6: ha_proxy_z1/0
(vm-643f3c80-93cb-4ccf-9239-1f85a407a317) is not responding.
1. Ignore problem
2. Reboot VM
3. Recreate VM using last known apply spec
4. Delete VM reference (DANGEROUS!)
Please choose a resolution [1 - 4]: 2
Below is the list of resolutions you've provided
Please make sure everything is fine and confirm your changes
1. Unknown VM (vm-e4da0933-52ba-473f-903d-a9ee09d1671f) is not
responding
Reboot VM
2. hm9000_z1/0 (vm-5ea4c90d-247e-43ad-a189-ff4d2d781854) is not
responding
Reboot VM
3. router_z1/0 (vm-c36efd49-7ac7-4b90-9779-b5192408e4a6) is not
responding
Reboot VM
4. nfs_z1/0 (vm-868e2bb9-ac61-49f0-86fb-38d5c338201b) is not
responding
Reboot VM
5. router_z2/0 (vm-773e83ee-c97f-4aa4-b163-d09a703a4678) is not
responding
Reboot VM
6. ha_proxy_z1/0 (vm-643f3c80-93cb-4ccf-9239-1f85a407a317) is not
responding
Reboot VM
Apply resolutions? (type 'yes' to continue): yes
Applying resolutions...
Director task 142
Started applying problem resolutions
Started applying problem resolutions > unresponsive_agent 154:
Reboot VM. Failed: Agent is responding now, skipping resolution (00:00:00)
Started applying problem resolutions > unresponsive_agent 184:
Reboot VM. Failed: Agent is responding now, skipping resolution (00:00:00)
Started applying problem resolutions > unresponsive_agent 177:
Reboot VM. Done (00:00:30)
Started applying problem resolutions > unresponsive_agent 179:
Reboot VM. Failed: Agent is responding now, skipping resolution (00:00:00)
Started applying problem resolutions > unresponsive_agent 185:
Reboot VM. Done (00:00:11)
Started applying problem resolutions > unresponsive_agent 180:
Reboot VM. Done (00:00:29)
Failed applying problem resolutions (00:01:10)
Task 142 done
Started 2015-06-12 20:04:01 UTC
Finished 2015-06-12 20:05:11 UTC
Duration 00:01:10
Cloudcheck is finished
Network section in cf-stub.yml
---
name: cf
director_uuid: b9a1bf7b-952f-48e1-a496-f6543d7a782c
releases:
- {name: cf, version: latest}
networks:
- name: cf1
subnets:
- range: 10.195.166.0/23
gateway: 10.195.166.1
static:
* - 10.195.166.105 - 10.195.166.135*
reserved:
# .1 is special
- 10.195.166.2 - 10.195.166.104
- 10.195.166.188 - 10.195.166.254
# .255 is special
dns: [10.166.168.183]
cloud_properties:
name: '10.195.166.x'
- name: cf2
subnets:
- range: 10.195.166.0/23
gateway: 10.195.166.1
static:
*- 10.195.166.136 - 10.195.166.166*
reserved:
# .1 is special
- 10.195.166.2 - 10.195.166.104
- 10.195.166.188 - 10.195.166.254
# .255 is special
dns: [10.166.168.183]
cloud_properties:
name: '10.195.166.x'
Thank you
From: Gwenn Etourneau <getourneau(a)pivotal.io>
Reply-To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Date: Monday, June 8, 2015 at 6:16 PM
To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Subject: Re: [cf-bosh] cf-stub.yml example with minimum or required
info
Try to make bosh cck before bosh deploy.
On Tue, Jun 9, 2015 at 3:43 AM, Ahmed Ali (ahmeali) <
ahmeali(a)cisco.com> wrote:
I was able to fix this by copying from the generated
cf-deployment.yml and use it in cf-stub.yml after changing cf release and
jobs section.
Now "bosh deploy" works but not complete, running into an issue I
think has to do with jobs/nodes misconfigured, not sure what are the list
of jobs/nodes I need to include and the properties of each one, any idea
where I can find information about the list of jobs/nodes and properties
for each such as templates/network ..etc?
"Bosh vms” shows some Vms in “unresponsive agent” and unknown job.
+------------------------------------+--------------------+---------------+----------------+
| Job/index | State | Resource
Pool | IPs |
+------------------------------------+--------------------+---------------+----------------+
| unknown/unknown | unresponsive agent |
| |
| unknown/unknown | unresponsive agent |
| |
| unknown/unknown | unresponsive agent |
| |
| unknown/unknown | unresponsive agent |
| |
| unknown/unknown | unresponsive agent |
| |
| unknown/unknown | unresponsive agent |
| |
| unknown/unknown | running |
medium_z1 | 10.195.166.106 |
| unknown/unknown | running |
router_z1 | 10.195.166.108 |
| unknown/unknown | running | small_z1
| 10.195.166.141 |
| api_z1/0 | running | large_z1
| 10.195.166.139 |
| api_z2/0 | running | large_z2
| 10.195.166.107 |
| clock_global/0 | running |
medium_z1 | 10.195.166.140 |
| consul_z1/0 | running |
medium_z1 | 10.195.166.122 |
| consul_z2/0 | running |
medium_z2 | 10.195.166.153 |
| etcd_z1/0 | running |
medium_z1 | 10.195.166.119 |
| etcd_z1/1 | running |
medium_z1 | 10.195.166.120 |
| etcd_z2/0 | running |
medium_z2 | 10.195.166.149 |
| hm9000_z1/0 | running |
medium_z1 | 10.195.166.142 |
| hm9000_z2/0 | running |
medium_z2 | 10.195.166.109 |
| loggregator_trafficcontroller_z1/0 | running | small_z1
| 10.195.166.145 |
| loggregator_trafficcontroller_z2/0 | running | small_z2
| 10.195.166.112 |
| loggregator_z1/0 | running |
medium_z1 | 10.195.166.144 |
| loggregator_z2/0 | running |
medium_z2 | 10.195.166.111 |
| login_z1/0 | running |
medium_z1 | 10.195.166.138 |
| postgres_z1/0 | running |
medium_z1 | 10.195.166.117 |
| routing_api_z1/0 | running |
medium_z1 | 10.195.166.121 |
| routing_api_z2/0 | running |
medium_z2 | 10.195.166.152 |
| runner_z1/0 | running |
runner_z1 | 10.195.166.143 |
| runner_z2/0 | running |
runner_z2 | 10.195.166.110 |
| stats_z1/0 | running | small_z1
| 10.195.166.136 |
| uaa_z1/0 | running |
medium_z1 | 10.195.166.137 |
| uaa_z2/0 | running |
medium_z2 | 10.195.166.105 |
+------------------------------------+--------------------+---------------+----------------+
VMs total: 32
Started preparing deployment > Binding existing deployment.
Failed: Timed out sending `get_state' to
eabc55f7-b402-42f7-85b5-e3732306b813 after 45 seconds (00:02:15)
Error 450002: Timed out sending `get_state' to
eabc55f7-b402-42f7-85b5-e3732306b813 after 45 seconds
Thanks
Ali
From: AHMED ALI <ahmeali(a)cisco.com>
Reply-To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Date: Friday, June 5, 2015 at 10:45 AM
To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Subject: Re: [cf-bosh] cf-stub.yml example with minimum or required
info
That fixed the error, thank you!
And run into another error :)
M-20JW:cf-release ali00$ bosh deployment cf-deployment.yml
Deployment set to
`/Users/ali00/deployments/cf-release/cf-deployment.yml'
M-20JW:cf-release ali00$ ./generate_deployment_manifest vsphere
cf-stub.yml > cf-deployment.yml
M-20JW:cf-release ali00$ bosh deploy
Processing deployment manifest
------------------------------
Getting deployment properties from director...
Compiling deployment manifest...
Please review all changes carefully
Deploying
---------
Deployment name: `cf-deployment.yml'
Director name: `bosh2'
Are you sure you want to deploy? (type 'yes' to continue): yes
Director task 38
Started unknown
Started unknown > Binding deployment. Done (00:00:00)
Error 140002: Template `consul_agent' (job `consul_z1') references
an unknown release `cf'
Task 38 error
For a more detailed error report, run: bosh task 38 --debug
M-20JW:cf-release ali00$
How can construct the section in cf-stub to provide correct cf
release which I think should be cf-210 in my case (not sure)?
Modified networks part in my cf-stub.yml
-------------------------------------------------------
releases:
- name: cf-210
version: latest
networks:
- name: cf1
subnets:
- range: 10.195.166.0/23
gateway: 10.195.166.1
static:
- 10.195.166.104 - 10.195.166.125
reserved:
# .1 is special
- 10.195.166.2 - 10.195.166.101
- 10.195.166.147 - 10.195.166.254
# .255 is special
dns: [10.166.168.183]
cloud_properties:
name: '10.195.166.x'
- name: cf2
subnets:
- range: 10.195.166.0/23
gateway: 10.195.166.1
static:
- 10.195.166.126 - 10.195.166.146
reserved:
# .1 is special
- 10.195.166.2 - 10.195.166.101
- 10.195.166.147 - 10.195.166.254
# .255 is special
dns: [10.166.168.183]
cloud_properties:
name: '10.195.166.x'
------------------------------------------------
Thanks
Ali
From: CF Runtime <cfruntime(a)gmail.com>
Reply-To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Date: Friday, June 5, 2015 at 10:11 AM
To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Subject: Re: [cf-bosh] cf-stub.yml example with minimum or required
info
Hi Ahmed,
It looks like you haven't allocated enough IPs in your network.
The line for reserved IPs "10.166.166.104 - 10.166.166.115" should
be increased to have at least 19 IPs. You'll need to decrease the number of
reserved addresses as well in order to increase the number of available IPs
in your network. We recommend "10.166.166.104 - 10.166.166.123"
for available IPs and "10.166.166.124 - 10.194.167.254" for your
reserved range. If you're tracking our current develop branch and not the
final releases you should look in
cf-release/spec/fixtures/vsphere/cf-stub.yml for the stub that we use to do
our vsphere acceptance tests.
Best,
Zachary Auerbach + Dan Lavine CF Runtime Team
On Thu, Jun 4, 2015 at 2:18 PM, Ahmed Ali (ahmeali) <
ahmeali(a)cisco.com> wrote:
Thanks Joseph for your help, please see the error below:
20JXXW:cf-release ali00$ ./generate_deployment_manifest vsphere
cf-stub.yml > cf-deployment.yml
2015/06/04 13:34:50 error generating manifest: unresolved nodes:
(( static_ips(12) )) in
./templates/cf-infrastructure-vsphere.yml
jobs.[5].networks.[0].static_ips
(( static_ips(16) )) in
./templates/cf-infrastructure-vsphere.yml
jobs.[8].networks.[0].static_ips
(( static_ips(14, 15) )) in
./templates/cf-infrastructure-vsphere.yml
jobs.[15].networks.[0].static_ips
(( static_ips(17, 18, 19) )) in
./templates/cf-infrastructure-vsphere.yml
jobs.[17].networks.[0].static_ips
(( jobs.postgres_z1.networks.cf1.static_ips.[0] )) in
dynaml properties.databases.address
(( properties.databases.address )) in dynaml
properties.ccdb.address
(( properties.databases.address )) in dynaml
properties.uaadb.address
M-2XX0JW:cf-release ali00$
I do not want to bug cf-bosh alias with every error I run into so
my ask is to find a sample of cf-stub.yml with all minimum required values,
Im sure Im missing a lot :), the sample online here
http://docs.cloudfoundry.org/deploying/cf-stub-vsphere.html, when
I first run it I got an error regarding “Error 40001: Required property
`range' was not specified in object”, then after I added “range” property I
got the error above.
Im looking for building a POC CF with minimum effort, do have one
network (10.166.166.0/23) and vSphere 5.x, I want to use it for
both CF networks (cf1 and cf2), not sure how many Ips I need on each
network, and if I have to specify nodes spec and vsphere info in cf-stub
since I do not see section for it?
I also tried bosh-lite and it worked fine on Ubuntu 14.
Here is my cf-stub.yml in case you want to have a look
# The following line helps maintain current documentation at
http://docs.cloudfoundry.org.
# code_snippet cf-stub-vsphere start
---
name: cloudfoundry
director_uuid: b9a1bf7b-952f-48e1-a496-f6543d7a782c
releases:
- name: cf-210
version: latest
networks:
- name: cf1
subnets:
- range: 10.166.166.0/23
gateway: 10.195.76.1
static:
- 10.166.166.104 - 10.166.166.115
reserved:
# .1 is special
- 10.166.166.2 - 10.166.166.101
- 10.166.166.120 - 10.194.167.254
# .255 is special
dns: [10.166.168.183]
cloud_properties:
name: '10.166.166.x'
- name: cf2
subnets:
- range: 10.166.166.0/23
gateway: 10.166.166.1
static:
- 10.166.166.120 - 10.166.166.140
reserved:
# .1 is special
- 10.166.166.2 - 10.166.166.101
- 10.166.166.120 - 10.195.167.254
# .255 is special
dns: [10.166.168.183]
cloud_properties:
name: '10.166.166.x'
jobs:
ha_proxy_z1:
properties:
ha_proxy:
disable_http: true
properties:
cc:
droplets:
droplet_directory_key: the_key
buildpacks:
buildpack_directory_key: bd_key
staging_upload_user: username
staging_upload_password: password
bulk_api_password: password
db_encryption_key: the_key
dea_next:
disk_mb: 2048
memory_mb: 1024
loggregator_endpoint:
shared_secret: loggregator_endpoint_secret
nats:
user: nats_user
password: nats_password
router:
enable_ssl: true
ssl_cert: |
-----BEGIN CERTIFICATE-----
MIIDBjCCAe4CCQCz3nn1SWrDdTANBgkqhkiG9w0BAQUFADBFMQswCQYDVQQGEwJB
VTETMBEGA1UECBMKU29tZS1TdGF0ZTEhMB8GA1UEChMYSW50ZXJuZXQgV2lkZ2l0
cyBQdHkgTHRkMB4XDTE1MDMwMzE4NTMyNloXDTE2MDMwMjE4NTMyNlowRTELMAkG
A1UEBhMCQVUxEzARBgNVBAgTClNvbWUtU3RhdGUxITAfBgNVBAoTGEludGVybmV0
IFdpZGdpdHMgUHR5IEx0ZDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AKtTK9xq/ycRO3fWbk1abunYf9CY6sl0Wlqm9UPMkI4j0itY2OyGyn1YuCCiEdM3
b8guGSWB0XSL5PBq33e7ioiaH98UEe+Ai+TBxnJsro5WQ/TMywzRDhZ4E7gxDBav
88ZY+y7ts0HznfxqEIn0Gu/UK+s6ajYcIy7d9L988+hA3K1FSdes8MavXhrI4xA1
fY21gESfFkD4SsqvrkISC012pa7oVw1f94slIVcAG+l9MMAkatBGxgWAQO6kxk5o
oH1Z5q2m0afeQBfFqzu5lCITLfgTWCUZUmbF6UpRhmD850/LqNtryAPrLLqXxdig
OHiWqvFpCusOu/4z1uGC5xECAwEAATANBgkqhkiG9w0BAQUFAAOCAQEAV5RAFVQy
8Krs5c9ebYRseXO6czL9/Rfrt/weiC1XLcDkE2i2yYsBXazMYr58o4hACJwe2hoC
bihBZ9XnVpASEYHDLwDj3zxFP/bTuKs7tLhP7wz0lo8i6k5VSPAGBq2kjc/cO9a3
TMmLPks/Xm42MCSWGDnCEX1854B3+JK3CNEGqSY7FYXU4W9pZtHPZ3gBoy0ymSpg
mpleiY1Tbn5I2X7vviMW7jeviB5ivkZaXtObjyM3vtPLB+ILpa15ZhDSE5o71sjA
jXqrE1n5o/GXHX+1M8v3aJc30Az7QAqWohW/tw5SoiSmVQZWd7gFht9vSzaH2WgO
LwcpBC7+cUJEww==
-----END CERTIFICATE-----
ssl_key: |
-----BEGIN RSA PRIVATE KEY-----
MIIEpAIBAAKCAQEAq1Mr3Gr/JxE7d9ZuTVpu6dh/0JjqyXRaWqb1Q8yQjiPSK1jY
7IbKfVi4IKIR0zdvyC4ZJYHRdIvk8Grfd7uKiJof3xQR74CL5MHGcmyujlZD9MzL
DNEOFngTuDEMFq/zxlj7Lu2zQfOd/GoQifQa79Qr6zpqNhwjLt30v3zz6EDcrUVJ
16zwxq9eGsjjEDV9jbWARJ8WQPhKyq+uQhILTXalruhXDV/3iyUhVwAb6X0wwCRq
0EbGBYBA7qTGTmigfVnmrabRp95AF8WrO7mUIhMt+BNYJRlSZsXpSlGGYPznT8uo
22vIA+ssupfF2KA4eJaq8WkK6w67/jPW4YLnEQIDAQABAoIBAQCDVqpcOoZKK9K8
Bt3eXQKEMJ2ji2cKczFFJ5MEm9EBtoJLCryZbqfSue3Fzpj9pBUEkBpk/4VT5F7o
0/Vmc5Y7LHRcbqVlRtV30/lPBPQ4V/eWtly/AZDcNsdfP/J1fgPSvaoqCr2ORLWL
qL/vEfyIeM4GcWy0+JMcPbmABslw9O6Ptc5RGiP98vCLHQh/++sOtj6PH1pt+2X/
Uecv3b1Hk/3Oe+M8ySorJD3KA94QTRnKX+zubkxRg/zCAki+as8rQc/d+BfVG698
ylUT5LVLNuwbWnffY2Zt5x5CDqH01mJnHmxzQEfn68rb3bGFaYPEn9EP+maQijv6
SsUM9A3lAoGBAODRDRn4gEIxjPICp6aawRrMDlRc+k6IWDF7wudjxJlaxFr2t7FF
rFYm+jrcG6qMTyq+teR8uHpcKm9X8ax0L6N6gw5rVzIeIOGma/ZuYIYXX2XJx5SW
SOas1xW6qEIbOMv+Xu9w2SWbhTgyRmtlxxjr2e7gQLz9z/vuTReJpInnAoGBAMMW
sq5lqUfAQzqxlhTobQ7tnB48rUQvkGPE92SlDj2TUt9phek2/TgRJT6mdcozvimt
JPhxKg3ioxG8NPmN0EytjpSiKqlxS1R2po0fb75vputfpw16Z8/2Vik+xYqNMTLo
SpeVkHu7fbtNYEK2qcU44OyOZ/V+5Oo9TuBIFRhHAoGACkqHhwDRHjaWdR2Z/w5m
eIuOvF3lN2MWZm175ouynDKDeoaAsiS2VttB6R/aRFxX42UHfoYXC8LcTmyAK5zF
8X3SMf7H5wtqBepQVt+Gm5zGSSqLcEnQ3H5c+impOh105CGoxt0rk4Ui/AeRIalv
C70AJOcvD3eu5aFq9gDe/1ECgYBAhkVbASzYGnMh+pKVH7rScSxto8v6/XBYT1Ez
7JOlMhD667/qvtFJtgIHkq7qzepbhnTv5x3tscQVnZY34/u9ILpD1s8dc+dibEvx
6S/gYLVorB5ois/DLMqaobRcew6Gs+XX9RPwmLahOJpZ9mh4XrOmCgPAYtP71YM9
ExpHCQKBgQCMMDDWGMRdFMJgXbx1uMere7OoniBdZaOexjbglRh1rMVSXqzBoU8+
yhEuHGAsHGWQdSBHnqRe9O0Bj/Vlw2VVEaJeL1ewRHb+jXSnuKclZOJgMsJAvgGm
SOWIahDrATA4g1T6yLBWQPhj3ZXD3eCMxT1Q3DvpG1DjgvXwmXQJAA==
-----END RSA PRIVATE KEY-----
cipher_suites:
TLS_RSA_WITH_RC4_128_SHA:TLS_RSA_WITH_AES_128_CBC_SHA
status:
user: router_user
password: router_password
login:
logout:
redirect:
parameter:
disable: false
uaa:
admin:
client_secret: admin_secret
batch:
username: batch_username
password: batch_password
cc:
client_secret: cc_client_secret
clients:
app-direct:
secret: app-direct_secret
developer_console:
secret: developer_console_secret
login:
secret: login_client_secret
notifications:
secret: notification_secret
doppler:
secret: doppler_secret
cloud_controller_username_lookup:
secret: cloud_controller_username_lookup_secret
gorouter:
secret: gorouter_secret
jwt:
verification_key: vk
signing_key: sk
scim:
users:
-
admin|fakepassword|scim.write,scim.read,openid,cloud_controller.admin,doppler.firehose
# code_snippet cf-stub-vsphere end
# The previous line helps maintain current documentation at
http://docs.cloudfoundry.org.
Thank you
Ahmed
From: CF Runtime <cfruntime(a)gmail.com>
Reply-To: "Discussions about the Cloud Foundry BOSH project." <
cf-bosh(a)lists.cloudfoundry.org>
Date: Wednesday, June 3, 2015 at 5:40 PM
To: "cf-bosh(a)lists.cloudfoundry.org" <
cf-bosh(a)lists.cloudfoundry.org>
Subject: Re: [cf-bosh] cf-stub.yml example with minimum or
required info
Hi Ali,
We try to keep those docs up to date, but it is possible they are
missing some pieces.
Can you tell me what errors you are getting?
Joseph Palermo
CF Runtime Team
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh
|
