Thanks Dmitriy – this fixed our issue J



From: Dmitriy Kalinin [mailto:dkalinin(a)pivotal.io]
Sent: 26 May 2015 19:54
To: Scott Taggart
Cc: CF BOSH Mailing List
Subject: Re: [cf-bosh] Most bosh director commands fail with a HTTP 500



There currently exists a problem in the Director during task cleanup.
Director tries to clean up task logs for the tasks that do not have
associated directory on disk.
https://www.pivotaltracker.com/story/show/95458780 will fix this.



To fix the Director until we release a bug fix:

- ssh as vcap into the Director VM

- run /var/vcap/jobs/director/bin/director_ctl console

- opens up console to the Director DB

- run Bosh::Director::Models::Task.where(output: nil).update(output:
'/tmp/123')

- updates tasks without task log directories to a dummy destination;
Director will be happy to run rm -rf /tmp/123 when it cleans up tasks.



After that you should be able to run `bosh vms` and other tasks again.





On Mon, May 25, 2015 at 2:27 PM, Scott Taggart <staggart(a)skyscapecloud.com>
wrote:

Hi folks,



One of my three bosh directors has gotten itself stuck in a strange state
where most (but not all) operations fail. I have recreated the director with
a couple of different stemcells (but same persistent disk) and the issue
persists. Looks like potentially a database issue on the director, but I
have done a very quick visual check of a few tables (e.g. vms, deployments)
and they seem fine from a glance... not sure what's going on.



Everything CF-related currently/previously under the director is continuing
to run fine in this AZ, it's just the director that's lost it:



$ bosh deployments

+---------------------+-----------------------+----------------------------------------------+--------------+
| Name | Release(s) | Stemcell(s) | Cloud Config |
+---------------------+-----------------------+----------------------------------------------+--------------+
| cf-mysql | cf-mysql/19 | bosh-vcloud-esxi-ubuntu-trusty-go_agent/2915 |
none |
+---------------------+-----------------------+----------------------------------------------+--------------+
| cf-services-contrib | cf-services-contrib/6 |
bosh-vcloud-esxi-ubuntu-trusty-go_agent/2915 | none |
+---------------------+-----------------------+----------------------------------------------+--------------+
| xxxxxxx_cf | cf/208 | bosh-vcloud-esxi-ubuntu-trusty-go_agent/2915 | none
|
+---------------------+-----------------------+----------------------------------------------+--------------+

Deployments total: 3



$ bosh releases

+---------------------+----------+-------------+
| Name | Versions | Commit Hash |
+---------------------+----------+-------------+
| cf | 208* | 5d00be54+ |
| cf-mysql | 19* | dfab036b+ |
| cf-services-contrib | 6* | 57fd2098+ |
+---------------------+----------+-------------+
(*) Currently deployed
(+) Uncommitted changes

Releases total: 3



$ bosh locks
No locks


$ bosh tasks
No running tasks



$ bosh vms
Deployment `cf-mysql'
HTTP 500:



$ bosh cloudcheck
Performing cloud check...

Processing deployment manifest
------------------------------
HTTP 500:



The relevant error I get from /var/vcap/sys/log/director/director.debug.log
on the director is:

E, [2015-05-25 21:20:15 #1010] [] ERROR -- Director: TypeError - no implicit
conversion of nil into String:
/var/vcap/packages/ruby/lib/ruby/2.1.0/fileutils.rb:1572:in `path'
/var/vcap/packages/ruby/lib/ruby/2.1.0/fileutils.rb:1572:in `block in
fu_list'
/var/vcap/packages/ruby/lib/ruby/2.1.0/fileutils.rb:1572:in `map'
/var/vcap/packages/ruby/lib/ruby/2.1.0/fileutils.rb:1572:in `fu_list'
/var/vcap/packages/ruby/lib/ruby/2.1.0/fileutils.rb:625:in `rm_r'
/var/vcap/packages/ruby/lib/ruby/2.1.0/fileutils.rb:654:in `rm_rf'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2957.0/lib/bosh/director/api/task_remover.rb:9:in
`block in remove'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/dataset/actions.rb:152:in
`block in each'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:525:in
`block (2 levels) in fetch_rows'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:720:in
`block in yield_hash_rows'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:714:in
`times'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:714:in
`yield_hash_rows'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:525:in
`block in fetch_rows'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:134:in
`execute'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:413:in
`_execute'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:242:in
`block (2 levels) in execute'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:425:in
`check_database_errors'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:242:in
`block in execute'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/database/connecting.rb:236:in
`block in synchronize'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/connection_pool/threaded.rb:104:in
`hold'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/database/connecting.rb:236:in
`synchronize'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:242:in
`execute'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/dataset/actions.rb:801:in
`execute'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/adapters/postgres.rb:525:in
`fetch_rows'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sequel-3.43.0/lib/sequel/dataset/actions.rb:152:in
`each'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2957.0/lib/bosh/director/api/task_remover.rb:8:in
`remove'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2957.0/lib/bosh/director/api/task_helper.rb:23:in
`create_task'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2957.0/lib/bosh/director/job_queue.rb:9:in
`enqueue'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2957.0/lib/bosh/director/api/vm_state_manager.rb:5:in
`fetch_vm_state'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2957.0/lib/bosh/director/api/controllers/deployments_controller.rb:182:in
`block in <class:DeploymentsController>'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1603:in
`block in compile!'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in
`[]'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in
`block (3 levels) in route!'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:985:in
`route_eval'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:966:in
`block (2 levels) in route!'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1006:in
`block in process_route'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in
`catch'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1004:in
`process_route'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:964:in
`block in route!'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in
`each'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:963:in
`route!'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1076:in
`block in dispatch!'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`block in invoke'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`catch'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`invoke'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1073:in
`dispatch!'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in
`block in call!'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`block in invoke'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`catch'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:1058:in
`invoke'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:898:in
`call!'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:886:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/xss_header.rb:18:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/path_traversal.rb:16:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/json_csrf.rb:18:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/base.rb:49:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-protection-1.5.3/lib/rack/protection/frame_options.rb:31:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-1.6.0/lib/rack/nulllogger.rb:9:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-1.6.0/lib/rack/head.rb:13:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:180:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:2014:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-1.6.0/lib/rack/urlmap.rb:66:in
`block in call'

/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-1.6.0/lib/rack/urlmap.rb:50:in
`each'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-1.6.0/lib/rack/urlmap.rb:50:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/rack-1.6.0/lib/rack/commonlogger.rb:33:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/sinatra-1.4.5/lib/sinatra/base.rb:217:in
`call'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/thin-1.5.1/lib/thin/connection.rb:81:in
`block in pre_process'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/thin-1.5.1/lib/thin/connection.rb:79:in
`catch'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/thin-1.5.1/lib/thin/connection.rb:79:in
`pre_process'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/thin-1.5.1/lib/thin/connection.rb:54:in
`process'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/thin-1.5.1/lib/thin/connection.rb:39:in
`receive_data'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in
`run_machine'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/eventmachine-1.0.3/lib/eventmachine.rb:187:in
`run'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/thin-1.5.1/lib/thin/backends/base.rb:63:in
`start'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/thin-1.5.1/lib/thin/server.rb:159:in
`start'
/var/vcap/packages/director/gem_home/ruby/2.1.0/gems/bosh-director-1.2957.0/bin/bosh-director:37:in
`<top (required)>'
/var/vcap/packages/director/bin/bosh-director:16:in `load'
/var/vcap/packages/director/bin/bosh-director:16:in `<main>'



I've wiped my local bosh config and re-targetted the director and tried
running bosh vms without specifying a deployment manifest (i.e. rule the
manifest out) - still get the same 500



Any tips appreciated!


Notice:
This message contains information that may be privileged or confidential and
is the property of Skyscape. It is intended only for the person to whom it
is addressed. If you are not the intended recipient, you are not authorised
to read, print, retain, copy, disseminate, distribute, or use this message
or any part thereof. If you receive this message in error, please notify the
sender immediately and delete all copies of this message. Skyscape reserves
the right to monitor all e-mail communications through its networks.
Skyscape Cloud Services Limited is registered in England and Wales: Company
No: 07619797. Registered office: Hartham Park, Hartham, Corsham, Wiltshire
SN13 0RP.

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh




______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


Notice:
This message contains information that may be privileged or confidential and is the property of Skyscape. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorised to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message. Skyscape reserves the right to monitor all e-mail communications through its networks. Skyscape Cloud Services Limited is registered in England and Wales: Company No: 07619797. Registered office: Hartham Park, Hartham, Corsham, Wiltshire SN13 0RP.

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________