Hi Cloud Foundry!
We are trying to split the cloud foundry deployment in to multiple
deployments. Each CF component will have its own deployment manifest.
We are doing this activity in an existing CF. We moved all components
except nats and etcd, into the new deployments. The original single
deployment is now having just these two jobs.

Of which, existing deployment is having 3 etcd machines. The migration
idea is to bring 4 new etcd machines in the cluster through new deployment.
Point all other components to these four etcd machines and delete the
existing 3 nodes.

However, if we delete the existing 3 nodes and do an update to form a 4
node cluster, the cluster breaks and as a result all running apps are going
down. (Because the canary job brings one node down for the update, as a
result tolerance is breached.)

We also tried to remove these three nodes from the cluster using etcdctl
command and tried to update deletion to the new deployment through bosh.
This also makes the bosh deployment to fail (etcd job is failing saying
"unequal number of nodes").

In this situation, what would be the best way to reduce the nodes in the
etcd cluster?

regards,
Surendhar