BOSH multi-zone resurrector


Aleksey Zalesov
 

Hello!

I have a question about resurrection algorithm BOSH uses in multi zone
deployment. Suppose we have three availability zones: z1, z2 and z3. We have
data/0 vm in z1 with resurrection enabled.

Suppose now that z1 goes down.

Will resurrector start new data/0 instance in another zone? If yes, how will
it chose between z2 and z3?

What if BOSH director itself is deployed to the z1?

Aleksey Zalesov,
Altoros



--
View this message in context: http://cf-bosh.70367.x6.nabble.com/BOSH-multi-zone-resurrector-tp526.html
Sent from the CF BOSH mailing list archive at Nabble.com.


Dmitriy Kalinin
 

Currently Director will bring back VMs in the same AZ as configured in the
manifest because resource pool is configured to point to that specific AZ.

If Director is in the affected zone you will have to redeploy Director to a
different AZ and restore DB from a backup or point to external DB that it
was using.



On Mon, Jul 27, 2015 at 12:28 PM, lexsys <aleksey.zalesov(a)altoros.com>
wrote:

Hello!

I have a question about resurrection algorithm BOSH uses in multi zone
deployment. Suppose we have three availability zones: z1, z2 and z3. We
have
data/0 vm in z1 with resurrection enabled.

Suppose now that z1 goes down.

Will resurrector start new data/0 instance in another zone? If yes, how
will
it chose between z2 and z3?

What if BOSH director itself is deployed to the z1?

Aleksey Zalesov,
Altoros



--
View this message in context:
http://cf-bosh.70367.x6.nabble.com/BOSH-multi-zone-resurrector-tp526.html
Sent from the CF BOSH mailing list archive at Nabble.com.
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh


John Wong
 

Hi also interested in this topic.

Is there a plan to be able to fallback to other zones? This seems like a
good capacity management but from automated recovery this seems like chore.
What do you suggest?

Thanks.

John

On Fri, Jul 31, 2015 at 6:28 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

Currently Director will bring back VMs in the same AZ as configured in the
manifest because resource pool is configured to point to that specific AZ.

If Director is in the affected zone you will have to redeploy Director to
a different AZ and restore DB from a backup or point to external DB that it
was using.



On Mon, Jul 27, 2015 at 12:28 PM, lexsys <aleksey.zalesov(a)altoros.com>
wrote:

Hello!

I have a question about resurrection algorithm BOSH uses in multi zone
deployment. Suppose we have three availability zones: z1, z2 and z3. We
have
data/0 vm in z1 with resurrection enabled.

Suppose now that z1 goes down.

Will resurrector start new data/0 instance in another zone? If yes, how
will
it chose between z2 and z3?

What if BOSH director itself is deployed to the z1?

Aleksey Zalesov,
Altoros



--
View this message in context:
http://cf-bosh.70367.x6.nabble.com/BOSH-multi-zone-resurrector-tp526.html
Sent from the CF BOSH mailing list archive at Nabble.com.
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh

_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh


Dmitriy Kalinin
 

We are currently working on adding first class concept of AZ to bosh:
https://github.com/cloudfoundry/bosh-notes/blob/master/availability-zones.md

Once that's complete, we can definitely think more about how that should
affect the resurrector (I've added tbd item for this). Tricky part is about
instance with persistent disks since they cannot be moved to a different
AZ. For instances that do not have persistent disks I could see us making a
choice of moving them to a different AZ.

Few questions arise out of that:
- when should we try a different AZ? after a few failures in the assigned
AZ?
- should next bosh deploy put the instance back into a correct AZ.(and if
not that could mean that over time your whole deployment slowly moves into
a single AZ)
- what to do with instances with persistent disks? should we just throw
away the disks and move?

What do you think?

On Tue, Aug 4, 2015 at 2:50 PM, John Wong <gokoproject(a)gmail.com> wrote:

Hi also interested in this topic.

Is there a plan to be able to fallback to other zones? This seems like a
good capacity management but from automated recovery this seems like chore.
What do you suggest?

Thanks.

John

On Fri, Jul 31, 2015 at 6:28 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

Currently Director will bring back VMs in the same AZ as configured in
the manifest because resource pool is configured to point to that specific
AZ.

If Director is in the affected zone you will have to redeploy Director to
a different AZ and restore DB from a backup or point to external DB that it
was using.



On Mon, Jul 27, 2015 at 12:28 PM, lexsys <aleksey.zalesov(a)altoros.com>
wrote:

Hello!

I have a question about resurrection algorithm BOSH uses in multi zone
deployment. Suppose we have three availability zones: z1, z2 and z3. We
have
data/0 vm in z1 with resurrection enabled.

Suppose now that z1 goes down.

Will resurrector start new data/0 instance in another zone? If yes, how
will
it chose between z2 and z3?

What if BOSH director itself is deployed to the z1?

Aleksey Zalesov,
Altoros



--
View this message in context:
http://cf-bosh.70367.x6.nabble.com/BOSH-multi-zone-resurrector-tp526.html
Sent from the CF BOSH mailing list archive at Nabble.com.
_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh

_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh