Issues with bosh-init conversion from micro-cli on vsphere


WOHLSTADTER, RICHARD W [AG/1000] <richard.w.wohlstadter@...>
 

Hello,

Were currently trying to switch from using bosh micro cli to bosh-init. I created a new yml for our microbosh instance and copied over the bosh-deployments.yml file from the micro-cli deployment. As it started, I could see it create the state file using info from the bosh-deployment with respect to the persistent disk so all looked well. When it got to actually deploying the new vm, it failed with the following error below. I looked and the persistent disk had gotten deleted. I believe this issue is that before deleting the microbosh vm, it does not unattach the persistent disk. So when the vm gets deleted, it also deletes the attached disk. To test the theory, I fixed things up and before running the conversion, I first manually unattached the persistent disk. It then ran to completion without any issues. I’m thinking this is a bug for bosh-init specifically when working with vsphere (Assume its not an issue out in aws since the volumes can be marked to not get deleted on vm termination).

-Rich



Error:

Started deploying
Waiting for the agent on VM 'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'... Failed (00:00:09)
Deleting VM 'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'... Finished (00:00:09)
Creating VM for instance 'bosh/0' from stemcell 'sc-705c2f0e-f015-4512-91bd-5961d5557fed'... Finished (00:00:41)
Waiting for the agent on VM 'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a' to be ready... Finished (00:00:45)
Attaching disk 'disk-249a0bd3-128b-43a8-846c-336995ada9ee' to VM 'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a'... Failed (00:00:41)
Failed deploying (00:02:32)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Command 'deploy' failed:
Deploying:
Creating instance 'bosh/0':
Updating instance disks:
Updating disks:
Deploying disk:
Attaching disk in the cloud:
CPI 'attach_disk' method responded with error: CmdError{"type":"Bosh::Clouds::DiskNotFound","message":"Could not find disk with id 'disk-249a0bd3-128b-43a8-846c-336995ada9ee'","ok_to_retry":false}
This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.


Jacek Szlachta
 

+1, literally did the same thing today with same result.

On 18 November 2015 at 22:33, WOHLSTADTER, RICHARD W [AG/1000] <
richard.w.wohlstadter(a)monsanto.com> wrote:

Hello,

Were currently trying to switch from using bosh micro cli to bosh-init. I
created a new yml for our microbosh instance and copied over the
bosh-deployments.yml file from the micro-cli deployment. As it started, I
could see it create the state file using info from the bosh-deployment with
respect to the persistent disk so all looked well. When it got to actually
deploying the new vm, it failed with the following error below. I looked
and the persistent disk had gotten deleted. I believe this issue is that
before deleting the microbosh vm, it does not unattach the persistent
disk. So when the vm gets deleted, it also deletes the attached disk. To
test the theory, I fixed things up and before running the conversion, I
first manually unattached the persistent disk. It then ran to completion
without any issues. I’m thinking this is a bug for bosh-init specifically
when working with vsphere (Assume its not an issue out in aws since the
volumes can be marked to not get deleted on vm termination).

-Rich



Error:

Started deploying
Waiting for the agent on VM 'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'...
Failed (00:00:09)
Deleting VM 'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'... Finished
(00:00:09)
Creating VM for instance 'bosh/0' from stemcell
'sc-705c2f0e-f015-4512-91bd-5961d5557fed'... Finished (00:00:41)
Waiting for the agent on VM 'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a' to
be ready... Finished (00:00:45)
Attaching disk 'disk-249a0bd3-128b-43a8-846c-336995ada9ee' to VM
'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a'... Failed (00:00:41)
Failed deploying (00:02:32)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Command 'deploy' failed:
Deploying:
Creating instance 'bosh/0':
Updating instance disks:
Updating disks:
Deploying disk:
Attaching disk in the cloud:
CPI 'attach_disk' method responded with error:
CmdError{"type":"Bosh::Clouds::DiskNotFound","message":"Could not find disk
with id 'disk-249a0bd3-128b-43a8-846c-336995ada9ee'","ok_to_retry":false}


This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.




Dmitriy Kalinin
 

did you somehow get your persistent disks inside the vm folder. did you use storage vmotion?

vsphere cpi places disks into a different folder explicitly so that when it deletes vms vsphere doesnt delete associated disks. we noticed that sometimes when vsphere moves vms around it moves the persistent disk into a vm folder which is specifically associated to that vm. vsphere then when deleting the vm also deletes the disk.

this behaviour is not different between micro or bosh-init since they use exactly same cpi source code.

im thinking may be we can do some safety checks to see if vsphere somehow moved the persistent disk into vm folder and raise an error if its found before we delete the vm.

did you by any chnace note where the disk was at the time of deletion?

Sent from my iPhone

On Nov 18, 2015, at 3:45 PM, Jacek Szlachta <jacek.szlachta(a)gmail.com> wrote:

+1, literally did the same thing today with same result.

On 18 November 2015 at 22:33, WOHLSTADTER, RICHARD W [AG/1000] <richard.w.wohlstadter(a)monsanto.com> wrote:
Hello,

Were currently trying to switch from using bosh micro cli to bosh-init. I created a new yml for our microbosh instance and copied over the bosh-deployments.yml file from the micro-cli deployment. As it started, I could see it create the state file using info from the bosh-deployment with respect to the persistent disk so all looked well. When it got to actually deploying the new vm, it failed with the following error below. I looked and the persistent disk had gotten deleted. I believe this issue is that before deleting the microbosh vm, it does not unattach the persistent disk. So when the vm gets deleted, it also deletes the attached disk. To test the theory, I fixed things up and before running the conversion, I first manually unattached the persistent disk. It then ran to completion without any issues. I’m thinking this is a bug for bosh-init specifically when working with vsphere (Assume its not an issue out in aws since the volumes can be marked to not get deleted on vm termination).

-Rich



Error:

Started deploying
Waiting for the agent on VM 'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'... Failed (00:00:09)
Deleting VM 'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'... Finished (00:00:09)
Creating VM for instance 'bosh/0' from stemcell 'sc-705c2f0e-f015-4512-91bd-5961d5557fed'... Finished (00:00:41)
Waiting for the agent on VM 'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a' to be ready... Finished (00:00:45)
Attaching disk 'disk-249a0bd3-128b-43a8-846c-336995ada9ee' to VM 'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a'... Failed (00:00:41)
Failed deploying (00:02:32)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Command 'deploy' failed:
Deploying:
Creating instance 'bosh/0':
Updating instance disks:
Updating disks:
Deploying disk:
Attaching disk in the cloud:
CPI 'attach_disk' method responded with error: CmdError{"type":"Bosh::Clouds::DiskNotFound","message":"Could not find disk with id 'disk-249a0bd3-128b-43a8-846c-336995ada9ee'","ok_to_retry":false}


This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.


Jacek Szlachta
 

Disk was in a different folder than the vm.

Actually, I cloned the vm for backup before upgrading with bosh-init - had
to detach the disk from the vm before cloning and then reattach again. At
the point of reattaching the disk was in a different folder. I have more
environments to upgrade, will keep an eye next time and confirm.

On 19 November 2015 at 00:06, Dmitriy Kalinin <dkalinin(a)pivotal.io> wrote:

did you somehow get your persistent disks inside the vm folder. did you
use storage vmotion?

vsphere cpi places disks into a different folder explicitly so that when
it deletes vms vsphere doesnt delete associated disks. we noticed that
sometimes when vsphere moves vms around it moves the persistent disk into a
vm folder which is specifically associated to that vm. vsphere then when
deleting the vm also deletes the disk.

this behaviour is not different between micro or bosh-init since they use
exactly same cpi source code.

im thinking may be we can do some safety checks to see if vsphere somehow
moved the persistent disk into vm folder and raise an error if its found
before we delete the vm.

did you by any chnace note where the disk was at the time of deletion?

Sent from my iPhone

On Nov 18, 2015, at 3:45 PM, Jacek Szlachta <jacek.szlachta(a)gmail.com>
wrote:

+1, literally did the same thing today with same result.

On 18 November 2015 at 22:33, WOHLSTADTER, RICHARD W [AG/1000] <
richard.w.wohlstadter(a)monsanto.com> wrote:

Hello,

Were currently trying to switch from using bosh micro cli to bosh-init.
I created a new yml for our microbosh instance and copied over the
bosh-deployments.yml file from the micro-cli deployment. As it started, I
could see it create the state file using info from the bosh-deployment with
respect to the persistent disk so all looked well. When it got to actually
deploying the new vm, it failed with the following error below. I looked
and the persistent disk had gotten deleted. I believe this issue is that
before deleting the microbosh vm, it does not unattach the persistent
disk. So when the vm gets deleted, it also deletes the attached disk. To
test the theory, I fixed things up and before running the conversion, I
first manually unattached the persistent disk. It then ran to completion
without any issues. I’m thinking this is a bug for bosh-init specifically
when working with vsphere (Assume its not an issue out in aws since the
volumes can be marked to not get deleted on vm termination).

-Rich



Error:

Started deploying
Waiting for the agent on VM
'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'... Failed (00:00:09)
Deleting VM 'vm-ae27499d-2f96-47c1-9315-ed2521fc71fe'... Finished
(00:00:09)
Creating VM for instance 'bosh/0' from stemcell
'sc-705c2f0e-f015-4512-91bd-5961d5557fed'... Finished (00:00:41)
Waiting for the agent on VM 'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a'
to be ready... Finished (00:00:45)
Attaching disk 'disk-249a0bd3-128b-43a8-846c-336995ada9ee' to VM
'vm-111e6a5e-427a-4d81-8675-ba95fa5e6f8a'... Failed (00:00:41)
Failed deploying (00:02:32)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Command 'deploy' failed:
Deploying:
Creating instance 'bosh/0':
Updating instance disks:
Updating disks:
Deploying disk:
Attaching disk in the cloud:
CPI 'attach_disk' method responded with error:
CmdError{"type":"Bosh::Clouds::DiskNotFound","message":"Could not find disk
with id 'disk-249a0bd3-128b-43a8-846c-336995ada9ee'","ok_to_retry":false}


This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.




WOHLSTADTER, RICHARD W [AG/1000] <richard.w.wohlstadter@...>
 

Yes, our persistent disks are located in a seperate folder away from the vm’s. I did it too seperate times to confirm that if attached, even though the disk is in a seperate folder, it will delete it along with the vm. Seems like as long as its attached to vm, doesnt matter where the disk is located.

-Rich
This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.


Dmitriy Kalinin
 

which vsphere version are you using?

also if possible can you provide a sanitized debug log from bosh-init (export BOSH_INIT_LOG_LEVEL=debug)?

we are looking mostly for output around cpi calls to look at cpi command output.

Sent from my iPhone

On Nov 19, 2015, at 9:08 AM, WOHLSTADTER, RICHARD W [AG/1000] <richard.w.wohlstadter(a)monsanto.com> wrote:

Yes, our persistent disks are located in a seperate folder away from the vm’s. I did it too seperate times to confirm that if attached, even though the disk is in a seperate folder, it will delete it along with the vm. Seems like as long as its attached to vm, doesnt matter where the disk is located.

-Rich


This e-mail message may contain privileged and/or confidential information, and is intended to be received only by persons entitled
to receive such information. If you have received this e-mail in error, please notify the sender immediately. Please delete it and
all attachments from any servers, hard drives or any other media. Other use of this e-mail by you is strictly prohibited.

All e-mails and attachments sent and received are subject to monitoring, reading and archival by Monsanto, including its
subsidiaries. The recipient of this e-mail is solely responsible for checking for the presence of "Viruses" or other "Malware".
Monsanto, along with its subsidiaries, accepts no liability for any damage caused by any such code transmitted by or accompanying
this e-mail or any attachment.


The information contained in this email may be subject to the export control laws and regulations of the United States, potentially
including but not limited to the Export Administration Regulations (EAR) and sanctions regulations issued by the U.S. Department of
Treasury, Office of Foreign Asset Controls (OFAC). As a recipient of this information you are obligated to comply with all
applicable U.S. export laws and regulations.


Rich Wohlstadter
 

Ok, I think I know what happened. I setup a brand new vm with micro cli and converted using bosh-init and it worked perfectly. I dug through the debug logs on how it works and got to thinking about disk modes on the persistent disks. A few months back we migrated datastores and I storage vmotioned the persistent disks, renamed them back to their original names and moved to original folders. It appears that when they moved, though, the disk mode got changed on them to be dependent (vs. independent/persistant). I tested again by changing the disk mode to dependent and sure enough it got deleted when the vm was deleted. Think the mystery is solved on my issue. Thanks for the tip on debug mode. Very useful.

-Rich


Dmitriy Kalinin
 

sounds like vsphere cpi can do a better job at preventing this. i ll drop in a story about stopping vm deletion and raising error in certain cases to be more paranoid.

Sent from my iPhone

On Nov 20, 2015, at 6:23 AM, Rich Wohlstadter <lethwin(a)gmail.com> wrote:

Ok, I think I know what happened. I setup a brand new vm with micro cli and converted using bosh-init and it worked perfectly. I dug through the debug logs on how it works and got to thinking about disk modes on the persistent disks. A few months back we migrated datastores and I storage vmotioned the persistent disks, renamed them back to their original names and moved to original folders. It appears that when they moved, though, the disk mode got changed on them to be dependent (vs. independent/persistant). I tested again by changing the disk mode to dependent and sure enough it got deleted when the vm was deleted. Think the mystery is solved on my issue. Thanks for the tip on debug mode. Very useful.

-Rich


Tom Sherrod <tom.sherrod@...>
 

+1 for being more paranoid.

On Fri, Nov 20, 2015 at 10:37 AM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

sounds like vsphere cpi can do a better job at preventing this. i ll drop
in a story about stopping vm deletion and raising error in certain cases to
be more paranoid.

Sent from my iPhone

On Nov 20, 2015, at 6:23 AM, Rich Wohlstadter <lethwin(a)gmail.com> wrote:

Ok, I think I know what happened. I setup a brand new vm with micro cli
and converted using bosh-init and it worked perfectly. I dug through the
debug logs on how it works and got to thinking about disk modes on the
persistent disks. A few months back we migrated datastores and I storage
vmotioned the persistent disks, renamed them back to their original names
and moved to original folders. It appears that when they moved, though,
the disk mode got changed on them to be dependent (vs.
independent/persistant). I tested again by changing the disk mode to
dependent and sure enough it got deleted when the vm was deleted. Think
the mystery is solved on my issue. Thanks for the tip on debug mode. Very
useful.

-Rich