On Nov 18, 2015, at 3:45 PM, Jacek Szlachta <jacek.szlachta(a)gmail.com> wrote:

+1, literally did the same thing today with same result.

On 18 November 2015 at 22:33, WOHLSTADTER, RICHARD W [AG/1000] <richard.w.wohlstadter(a)monsanto.com> wrote:
Hello,

Were currently trying to switch from using bosh micro cli to bosh-init. I created a new yml for our microbosh instance and copied over the bosh-deployments.yml file from the micro-cli deployment. As it started, I could see it create the state file using info from the bosh-deployment with respect to the persistent disk so all looked well. When it got to actually deploying the new vm, it failed with the following error below. I looked and the persistent disk had gotten deleted. I believe this issue is that before deleting the microbosh vm, it does not unattach the persistent disk. So when the vm gets deleted, it also deletes the attached disk. To test the theory, I fixed things up and before running the conversion, I first manually unattached the persistent disk. It then ran to completion without any issues. I’m thinking this is a bug for bosh-init specifically when working with vsphere (Assume its not an issue out in aws since the volumes can be marked to not get deleted on vm termination).

-Rich



Error:

Started deploying
Waiting for the agent on VM 'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'... Failed (00:00:09)
Deleting VM 'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'... Finished (00:00:09)
Creating VM for instance 'bosh/0' from stemcell 'sc-705c2f0e-f015-4512-91bd-5961d5557fed'... Finished (00:00:41)
Waiting for the agent on VM 'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a' to be ready... Finished (00:00:45)
Attaching disk 'disk-249a0bd3-128b-43a8-846c-336995ada9ee' to VM 'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a'... Failed (00:00:41)
Failed deploying (00:02:32)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Command 'deploy' failed:
Deploying:
Creating instance 'bosh/0':
Updating instance disks:
Updating disks:
Deploying disk:
Attaching disk in the cloud:
CPI 'attach_disk' method responded with error: CmdError{"type":"Bosh::Clouds::DiskNotFound","message":"Could not find disk with id 'disk-249a0bd3-128b-43a8-846c-336995ada9ee'","ok_to_retry":false}


This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.

did you somehow get your persistent disks inside the vm folder. did you
use storage vmotion?

vsphere cpi places disks into a different folder explicitly so that when
it deletes vms vsphere doesnt delete associated disks. we noticed that
sometimes when vsphere moves vms around it moves the persistent disk into a
vm folder which is specifically associated to that vm. vsphere then when
deleting the vm also deletes the disk.

this behaviour is not different between micro or bosh-init since they use
exactly same cpi source code.

im thinking may be we can do some safety checks to see if vsphere somehow
moved the persistent disk into vm folder and raise an error if its found
before we delete the vm.

did you by any chnace note where the disk was at the time of deletion?

Sent from my iPhone

On Nov 18, 2015, at 3:45 PM, Jacek Szlachta <jacek.szlachta(a)gmail.com>
wrote:

+1, literally did the same thing today with same result.

On 18 November 2015 at 22:33, WOHLSTADTER, RICHARD W [AG/1000] <
richard.w.wohlstadter(a)monsanto.com> wrote:

Hello,

Were currently trying to switch from using bosh micro cli to bosh-init.
I created a new yml for our microbosh instance and copied over the
bosh-deployments.yml file from the micro-cli deployment. As it started, I
could see it create the state file using info from the bosh-deployment with
respect to the persistent disk so all looked well. When it got to actually
deploying the new vm, it failed with the following error below. I looked
and the persistent disk had gotten deleted. I believe this issue is that
before deleting the microbosh vm, it does not unattach the persistent
disk. So when the vm gets deleted, it also deletes the attached disk. To
test the theory, I fixed things up and before running the conversion, I
first manually unattached the persistent disk. It then ran to completion
without any issues. I’m thinking this is a bug for bosh-init specifically
when working with vsphere (Assume its not an issue out in aws since the
volumes can be marked to not get deleted on vm termination).

-Rich



Error:

Started deploying
Waiting for the agent on VM
'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'... Failed (00:00:09)
Deleting VM 'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'... Finished
(00:00:09)
Creating VM for instance 'bosh/0' from stemcell
'sc-705c2f0e-f015-4512-91bd-5961d5557fed'... Finished (00:00:41)
Waiting for the agent on VM 'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a'
to be ready... Finished (00:00:45)
Attaching disk 'disk-249a0bd3-128b-43a8-846c-336995ada9ee' to VM
'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a'... Failed (00:00:41)
Failed deploying (00:02:32)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Command 'deploy' failed:
Deploying:
Creating instance 'bosh/0':
Updating instance disks:
Updating disks:
Deploying disk:
Attaching disk in the cloud:
CPI 'attach_disk' method responded with error:
CmdError{"type":"Bosh::Clouds::DiskNotFound","message":"Could not find disk
with id 'disk-249a0bd3-128b-43a8-846c-336995ada9ee'","ok_to_retry":false}


This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.

On Nov 19, 2015, at 9:08 AM, WOHLSTADTER, RICHARD W [AG/1000] <richard.w.wohlstadter(a)monsanto.com> wrote:

Yes, our persistent disks are located in a seperate folder away from the vm’s. I did it too seperate times to confirm that if attached, even though the disk is in a seperate folder, it will delete it along with the vm. Seems like as long as its attached to vm, doesnt matter where the disk is located.

-Rich


This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.

On Nov 20, 2015, at 6:23 AM, Rich Wohlstadter <lethwin(a)gmail.com> wrote:

Ok, I think I know what happened. I setup a brand new vm with micro cli and converted using bosh-init and it worked perfectly. I dug through the debug logs on how it works and got to thinking about disk modes on the persistent disks. A few months back we migrated datastores and I storage vmotioned the persistent disks, renamed them back to their original names and moved to original folders. It appears that when they moved, though, the disk mode got changed on them to be dependent (vs. independent/persistant). I tested again by changing the disk mode to dependent and sure enough it got deleted when the vm was deleted. Think the mystery is solved on my issue. Thanks for the tip on debug mode. Very useful.

-Rich