Re: Independent AZs deployments vs. streched deployments


David Sabeti
 

I just thought I'd point out that cf-deployment doesn't include etcd any longer. It also includes ops-files for deploying the experimental BOSH DNS release, which obviates the need for consul as well -- depending on your aversion to risk, or lack thereof, you may consider getting rid of consul using those ops-files. Alternatively, you could deploy consul to two AZs in the same way that Marco described deploying MySQL -- one consul node in z1, and two nodes in z2.

Are you using cf-mysql as the internal database? If so, that's the last component in that uses three AZs, and it sounds like Marco has some options for you there.

On Fri, Jan 12, 2018 at 7:44 AM Marco Nicosia <mnicosia@...> wrote:
I can’t speak for Consul nor Etcd, but cf-MySQL, in HA mode, can be deployed in a 2+1 mode where you’re still deploying an odd number of VMs, for consensus, but only deployed to two AZs.

This gives you some degree of protection to AZ failure, but yes, not the same resiliency as three.

Let’s say you have two AZs, the “major,” with two cluster nodes, and the “minor” with only one.

With two AZs, if there is a net split, the AZ with the majority of cluster nodes will continue to run.

If you suffer an AZ failure, however, you run a 50/50 chance of staying up. If the major AZ goes down, you’re down. In that respect you’re no better than running a single AZ. However you may have a faster path to recovery, in that you will have a seed to rebuild in the minor AZ possibly more quickly than rebuilding the major AZ.

This differs from traditional HA where there are two equal master nodes, one in each AZ. This does give the ability to fail back and forth between the two, but typically not automatically.

If you are interested in investing some effort, we could probably work together to establish the processes by which one could “recover” the minor AZ to run in non-HA mode. I think it should be possible to write yourself a “break glass in case of emergency” manifest which you could use to redeploy the minor AZ to restore availability.


On Fri, Jan 12, 2018 at 05:12 Juan Pablo Genovese <juanpgenovese@...> wrote:
Matthias, Maxim,

thank you for your input.
In my experience, a stretched architecture is the way to go, I agree on that, but sadly, we are limited to two and only two AZs. The only good thing is that the network connection between them is outstanding.
Just like Maxim said, the problem would be with Consul and etcd, which has to be deployed to odd numbers so the split brain problem can be avoided. For what I know, Consul's consensus is achieved on a (n/2) + 1 active members, so deploying 2+3 Consul nodes won't work.
I wonder if anybody solved this issue or managed to provide a workaround without individual CF deployments.



2018-01-12 11:05 GMT+00:00 Matthias Winzeler <matthias.winzeler@...>:
Hi Juan-Pablo

As long as your AZs are connected using a low-latency network (i.e. within the same data center), you can happily deploy a CF over multiple zones.
This is fully supported by CF/BOSH (from an operator perspective, it makes no difference whether you use multiple zones or 1) and transparent to your users (their apps will be evenly distributed over all zones).

We have around 20 CF installations which all run with 3 zones each. Downtime/maintenance of a zone is not affecting the end users and it's easy to manage it from an ops perspective.

We never tried the data synchronization approach since this requires manual plumbing and brings some additional challenges (i.e. blobstores and backing databases would need to be synchronized).

I think the 1 CF : n AZs setup is also what Pivotal recommends, i.e. for AWS use multiple AZs, for VCenter use multiple clusters.

Best regards
Matthias
Swisscom Application Cloud

2018-01-12 11:33 GMT+01:00 Juan Pablo Genovese <juanpgenovese@...>:
Hello everyone!

I'm having an internal discussion about what is the best way to deploy CF in two different AZs.
One proposal is to deploy two CF foundations, one per each AZ, and synchronize data.
The other proposal is one foundation encompassing both AZs.

Each one of us has its own personal views, but I'm very curious about past experiences and pains you might have had experienced with both models.

Thank you!

--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com




--




--
Mis mejores deseos,
Best wishes,
Meilleurs vœux,

Juan Pablo
------------------------------------------------------
http://www.jpgenovese.com

--
--
  Marco Nicosia
  Product Manager
  Pivotal Software, Inc.

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.