bosh persistent datatore migration


Jo Stockley
 

Hi,
I have BOSH deployed into vSphere and it is configured to use datastore1 for persistent data. I now need to use datastore2 and move all persistent data from datastore1 to datastore2.

I assume I only need to modify the deployment manifest for BOSH itself and change persistent_datastore_pattern from datastore1 to datastore2 and re-deploy.

I've searched but cant find any documentation that explains this and how the deployed apps get there persistent disks moved.
DO they need to be redeployed, recreated or what?

Any help gratefully received!

Thanks,
Jo


Dmitriy Kalinin
 

I dont believe current vsphere CPI provides an option for that, though it's
definitely possible to implement. One way it could be implemented is with
an additional CPI config that would list datastores that should be migrated
off. Doing bosh deploy --recreate then would reattach the disks and CPI
could make sure that disks should be moved to wanted datastores.

Alternatively I believe some people have moved their datastore contents via
the regular vsphere datastore copy while shutting off deployments.

On Tue, Jun 30, 2015 at 2:00 PM, Stockley, Jonathan <
jonathan.stockley(a)emc.com> wrote:

Hi,

I have BOSH deployed into vSphere and it is configured to use datastore1
for persistent data. I now need to use datastore2 and move all persistent
data from datastore1 to datastore2.



I assume I only need to modify the deployment manifest for BOSH itself and
change persistent_datastore_pattern from datastore1 to datastore2 and
re-deploy.



I’ve searched but cant find any documentation that explains this and how
the deployed apps get there persistent disks moved.

DO they need to be redeployed, recreated or what?



Any help gratefully received!



Thanks,

Jo







_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh


Jo Stockley
 

Thanks for the response, Dmitriy,
I did some digging and it does appear that the vSphere CPI supports this. I'm using BOSH version 1.2732 and the vSphere CPI has a Client#move_disk method in client.rb that is called by Cloud#attach_disk method in cloud.rb.

To test I changed the persistent_datastore_pattern property from datastore1 to datastore2 and re-deployed to update the BOSH.
The I did a deploy --recreate on my application and it worked, the persistent data files for each job were moved to datastore2.
However, I ran into a problem with a job that has 500GB persistent data. The move took more than one hour and BOSH timed out with the message:
Error 100: Task taking too long (1 hour)
I looked through the bosh task --debug output and found that the Client#wait_for_task method only waits for CPI tasks for up to 1 hour.
I changed this to 5 hour and now even the very large persistent disk gets moved successfully.

I will look into adding a vcenter property "task_timeout" that will default to 3600 seconds (as now) but will allow an override in the manifest for our use case.

Thanks again.

Jo

On Wed, 1 Jul 2015 at 3:23 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io> wrote:

I dont believe current vsphere CPI provides an option for that, though it's definitely possible to implement. One way it could be implemented is with an additional CPI config that would list datastores that should be migrated off. Doing bosh deploy --recreate then would reattach the disks and CPI could make sure that disks should be moved to wanted datastores.

Alternatively I believe some people have moved their datastore contents via the regular vsphere datastore copy while shutting off deployments.


Dmitriy Kalinin
 

You are using older version of BOSH that did support this. Since then we
have removed a database that vsphere cpi maintained to store disk paths
(but was misleading in other cases) and I do not believe this would have
worked in newer BOSH versions since disks would not be found in specified
datastores.

On Thu, Jul 2, 2015 at 3:08 PM, Stockley, Jonathan <
jonathan.stockley(a)emc.com> wrote:

Thanks for the response, Dmitriy,
I did some digging and it does appear that the vSphere CPI supports this.
I'm using BOSH version 1.2732 and the vSphere CPI has a Client#move_disk
method in client.rb that is called by Cloud#attach_disk method in cloud.rb.

To test I changed the persistent_datastore_pattern property from
datastore1 to datastore2 and re-deployed to update the BOSH.
The I did a deploy --recreate on my application and it worked, the
persistent data files for each job were moved to datastore2.
However, I ran into a problem with a job that has 500GB persistent data.
The move took more than one hour and BOSH timed out with the message:
Error 100: Task taking too long (1 hour)
I looked through the bosh task --debug output and found that the
Client#wait_for_task method only waits for CPI tasks for up to 1 hour.
I changed this to 5 hour and now even the very large persistent disk gets
moved successfully.

I will look into adding a vcenter property "task_timeout" that will
default to 3600 seconds (as now) but will allow an override in the manifest
for our use case.

Thanks again.

Jo

On Wed, 1 Jul 2015 at 3:23 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

I dont believe current vsphere CPI provides an option for that, though
it's definitely possible to implement. One way it could be implemented is
with an additional CPI config that would list datastores that should be
migrated off. Doing bosh deploy --recreate then would reattach the disks
and CPI could make sure that disks should be moved to wanted datastores.

Alternatively I believe some people have moved their datastore contents
via the regular vsphere datastore copy while shutting off deployments.
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh


Jo Stockley
 

Does this mean that, in latest BOSH release it is not possible to move disks from one datastore to another using BOSH?
If so, what, if any, are the plans to bring back support for doing this?

From: Dmitriy Kalinin [mailto:dkalinin(a)pivotal.io]
Sent: Thursday, July 02, 2015 3:13 PM
To: Discussions about the Cloud Foundry BOSH project.
Subject: Re: [cf-bosh] bosh persistent datatore migration

You are using older version of BOSH that did support this. Since then we have removed a database that vsphere cpi maintained to store disk paths (but was misleading in other cases) and I do not believe this would have worked in newer BOSH versions since disks would not be found in specified datastores.

On Thu, Jul 2, 2015 at 3:08 PM, Stockley, Jonathan <jonathan.stockley(a)emc.com<mailto:jonathan.stockley(a)emc.com>> wrote:
Thanks for the response, Dmitriy,
I did some digging and it does appear that the vSphere CPI supports this. I'm using BOSH version 1.2732 and the vSphere CPI has a Client#move_disk method in client.rb that is called by Cloud#attach_disk method in cloud.rb.

To test I changed the persistent_datastore_pattern property from datastore1 to datastore2 and re-deployed to update the BOSH.
The I did a deploy --recreate on my application and it worked, the persistent data files for each job were moved to datastore2.
However, I ran into a problem with a job that has 500GB persistent data. The move took more than one hour and BOSH timed out with the message:
Error 100: Task taking too long (1 hour)
I looked through the bosh task --debug output and found that the Client#wait_for_task method only waits for CPI tasks for up to 1 hour.
I changed this to 5 hour and now even the very large persistent disk gets moved successfully.

I will look into adding a vcenter property "task_timeout" that will default to 3600 seconds (as now) but will allow an override in the manifest for our use case.

Thanks again.

Jo

On Wed, 1 Jul 2015 at 3:23 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io<mailto:dkalinin(a)pivotal.io>> wrote:

I dont believe current vsphere CPI provides an option for that, though it's definitely possible to implement. One way it could be implemented is with an additional CPI config that would list datastores that should be migrated off. Doing bosh deploy --recreate then would reattach the disks and CPI could make sure that disks should be moved to wanted datastores.

Alternatively I believe some people have moved their datastore contents via the regular vsphere datastore copy while shutting off deployments.
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org<mailto:cf-bosh(a)lists.cloudfoundry.org>
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh


Dmitriy Kalinin
 

Correct.

I've entered a story to remedy that:
https://www.pivotaltracker.com/story/show/98603840. Description includes a
way that I think we can do that.

How do you feel about searching for a disk through all datastores in a
datacenter?

On Tue, Jul 7, 2015 at 2:30 PM, Stockley, Jonathan <
jonathan.stockley(a)emc.com> wrote:

Does this mean that, in latest BOSH release it is not possible to move
disks from one datastore to another using BOSH?

If so, what, if any, are the plans to bring back support for doing this?



*From:* Dmitriy Kalinin [mailto:dkalinin(a)pivotal.io]
*Sent:* Thursday, July 02, 2015 3:13 PM
*To:* Discussions about the Cloud Foundry BOSH project.
*Subject:* Re: [cf-bosh] bosh persistent datatore migration



You are using older version of BOSH that did support this. Since then we
have removed a database that vsphere cpi maintained to store disk paths
(but was misleading in other cases) and I do not believe this would have
worked in newer BOSH versions since disks would not be found in specified
datastores.



On Thu, Jul 2, 2015 at 3:08 PM, Stockley, Jonathan <
jonathan.stockley(a)emc.com> wrote:

Thanks for the response, Dmitriy,
I did some digging and it does appear that the vSphere CPI supports this.
I'm using BOSH version 1.2732 and the vSphere CPI has a Client#move_disk
method in client.rb that is called by Cloud#attach_disk method in cloud.rb.

To test I changed the persistent_datastore_pattern property from
datastore1 to datastore2 and re-deployed to update the BOSH.
The I did a deploy --recreate on my application and it worked, the
persistent data files for each job were moved to datastore2.
However, I ran into a problem with a job that has 500GB persistent data.
The move took more than one hour and BOSH timed out with the message:
Error 100: Task taking too long (1 hour)
I looked through the bosh task --debug output and found that the
Client#wait_for_task method only waits for CPI tasks for up to 1 hour.
I changed this to 5 hour and now even the very large persistent disk gets
moved successfully.

I will look into adding a vcenter property "task_timeout" that will
default to 3600 seconds (as now) but will allow an override in the manifest
for our use case.

Thanks again.

Jo

On Wed, 1 Jul 2015 at 3:23 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

I dont believe current vsphere CPI provides an option for that, though
it's definitely possible to implement. One way it could be implemented is
with an additional CPI config that would list datastores that should be
migrated off. Doing bosh deploy --recreate then would reattach the disks
and CPI could make sure that disks should be moved to wanted datastores.

Alternatively I believe some people have moved their datastore contents
via the regular vsphere datastore copy while shutting off deployments.
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh



_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh