restoring bosh deployment state failing


nshrest6@...
 

Hi ... i had a bosh director running which i updated with current vsphere cpi release 48, which failed due to the issue with ruby2.4, i tried reverting it back with old versions now i am running into issues .... 
```
Started deploying
  Waiting for the agent on VM 'vm-c3e42263-5167-467d-bda5-04e8762f63ec'... Failed (00:00:09)
  Deleting VM 'vm-c3e42263-5167-467d-bda5-04e8762f63ec'... Finished (00:00:08)
  Creating VM for instance 'bosh/0' from stemcell 'sc-5eae3672-c5cb-4351-8bf8-7972b464d0b4'... Finished (00:01:07)
  Waiting for the agent on VM 'vm-dc84d899-5444-483d-aeb6-1a247a04a56d' to be ready... Finished (00:00:27)
  Attaching disk 'disk-22e02c8a-b143-4534-a640-85705067887c' to VM 'vm-dc84d899-5444-483d-aeb6-1a247a04a56d'... Finished (00:00:18)
  Creating disk... Finished (00:00:07)
  Attaching disk 'disk-a8e62197-dfed-4596-b07e-4cf9686e852e' to VM 'vm-dc84d899-5444-483d-aeb6-1a247a04a56d'... Finished (00:00:18)
  Migrating disk content from 'disk-22e02c8a-b143-4534-a640-85705067887c' to 'disk-a8e62197-dfed-4596-b07e-4cf9686e852e'... Finished (00:01:57)
  Detaching disk 'disk-22e02c8a-b143-4534-a640-85705067887c'... Finished (00:00:10)
  Deleting disk 'disk-22e02c8a-b143-4534-a640-85705067887c'... Finished (00:00:04)
  Rendering job templates... Finished (00:00:06)
  Compiling package 'openjdk_1.8.0/a6b85c1cd75382025bbfa49abb737015575aec44'... Skipped [Package already compiled] (00:00:01)
  Compiling package 'ruby/c1086875b047d112e46756dcb63d8f19e63b3ac4'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'mysql/b7e73acc0bfe05f1c6cbfd97bf92d39b0d3155d5'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'libpq/826813f983d38b4b4a95bb8a3df1a2d0efab14b0'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'vsphere_cpi_ruby/14067294a0cd16a61646eedc3de4e9ed22d46076'... Finished (00:02:20)
  Compiling package 'credhub/c113daadcde5f2add56fb8f62313a96c6e98697e'... Skipped [Package already compiled] (00:00:01)
  Compiling package 'vsphere_cpi_mkisofs/72aac8fb0c0089065a00ef38a4e30d7d0e5a16ea'... Finished (00:02:44)
  Compiling package 'verify_multidigest/8fc5d654cebad7725c34bb08b3f60b912db7094a'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'lunaclient/b922e045db5246ec742f0c4d1496844942d6167a'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'bosh-gcscli/83d331c7b6d04de64cd5257a47e1e92021cb4c8a'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'postgres/3b1089109c074984577a0bac1b38018d7a2890ef'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'uaa_utils/20557445bf996af17995a5f13bf5f87000600f2e'... Skipped [Package already compiled] (00:00:00)
  Compiling package 's3cli/bb1c1976d221fdadf13a6bc873896cd5e2433580'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'pg_utils_9.4/dbd00a0758a5e6225e1121bfd444db6ec59204ee'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'davcli/5f08f8d5ab3addd0e11171f739f072b107b30b8c'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'director/ea00c83b4558293b1956564a4532e1af562ea6e0'... Skipped [Package already compiled] (00:00:01)
  Compiling package 'postgres-9.4/1da82648840de67015d379264846a447118261a7'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'nats/63ae42eb73527625307ff522fb402832b407321d'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'vsphere_cpi/e6c27f384060c8d0260f6f0310853d1a886b1128'... Finished (00:00:57)
  Compiling package 'nginx/57ca1d048957399c500e0f5fd3275ed4c6d4f762'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'mariadb_10.1.23/6ab14e132241110cff0dc160137b71a967d29d53'... Skipped [Package already compiled] (00:00:00)
  Compiling package 'uaa/33da697bb3343793c762f06970868565a71d053a'... Skipped [Package already compiled] (00:00:03)
  Compiling package 'health_monitor/aa43dacd332bda1131b141aada0ca45b4302273c'... Skipped [Package already compiled] (00:00:00)
  Updating instance 'bosh/0'... Finished (00:01:18)
  Waiting for instance 'bosh/0' to be running... Failed (00:06:00)
Failed deploying (00:18:44)
 
Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)
```
 
i logged to bosh director and monit process shows 
```
/:/var/vcap/sys/log# monit summary 
The Monit daemon 5.2.5 uptime: 5m 
 
Process 'nats'                      running
Process 'postgres'                  running
Process 'blobstore_nginx'           running
Process 'director'                  not monitored
Process 'worker_1'                  not monitored
Process 'worker_2'                  not monitored
Process 'worker_3'                  not monitored
Process 'director_scheduler'        running
Process 'director_nginx'            running
Process 'health_monitor'            running
Process 'uaa'                       running
Process 'credhub'                   Does not exist
System 'system_localhost'           running
```
 
i am confused where do i start troubleshooting ... any idea if someone encountered similar issue during the bosh director restore ?