Hi,

We installed CF210 recently (in single AZ).
Single etcd instance(instance number is 1) runs very well.
But multi etcd instances(instance number is 3) don't work well in single AZ.

And from the release Notes on http://bosh.io/releases/github.com/cloudfoundry/cf-release?version=210, there is one term I’m confused about:

“Upgrade etcd server to 2.0.1 details<https://www.pivotaltracker.com/story/show/91070214>
Should be run as 1 node (for small deployments) or 3 nodes spread across zones (for HA)”

May I ask it means:

“1. Should be run as 1 node (but can NOT be run as 3 nodes) in environment with only one zone.
2. Should be run as 3 nodes in environment with more than one zones. should spread them across zones.”

Or means:
“1. can be run as 1 node, (and can also be run as 3 nodes) in environment with only one zone.
2. Should be run as 3 nodes in environment with more than one zones. should spread them across zones.”

Thanks,

Tony
Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.


Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.


If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe(a)fast.au.fujitsu.com

I guess ,
It can run with 3 nodes but it seems it not doing well with the 3 nodes on
the same zone.



On Thu, Jul 16, 2015 at 10:18 AM, Li, Tony <Tonyl(a)fast.au.fujitsu.com>
wrote:

Hi,



We installed CF210 recently (in single AZ).

Single etcd instance(instance number is 1) runs very well.

But multi etcd instances(instance number is 3) don't work well in single
AZ.



And from the release Notes on
http://bosh.io/releases/github.com/cloudfoundry/cf-release?version=210,
there is one term I’m confused about:



“Upgrade etcd server to 2.0.1 details
<https://www.pivotaltracker.com/story/show/91070214>

Should be run as 1 node (for small deployments) or 3 nodes spread across
zones (for HA)”



May I ask it means:



“1. Should be run as 1 node (but can NOT be run as 3 nodes) in
environment with only one zone.

2. Should be run as 3 nodes in environment with more than one zones.
should spread them across zones.”



Or means:

“1. can be run as 1 node, (and can also be run as 3 nodes) in
environment with only one zone.

2. Should be run as 3 nodes in environment with more than one zones.
should spread them across zones.”



Thanks,



*Tony*
Disclaimer

The information in this e-mail is confidential and may contain content
that is subject to copyright and/or is commercial-in-confidence and is
intended only for the use of the above named addressee. If you are not the
intended recipient, you are hereby notified that dissemination, copying or
use of the information is strictly prohibited. If you have received this
e-mail in error, please telephone Fujitsu Australia Software Technology Pty
Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the
document and all copies thereof.

Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly
transmit a virus within an email communication, it is the receiver’s
responsibility to scan all communication and any files attached for
computer viruses and other defects. Fujitsu Australia Software Technology
Pty Ltd does not accept liability for any loss or damage (whether direct,
indirect, consequential or economic) however caused, and whether by
negligence or otherwise, which may result directly or indirectly from this
communication or any files attached.

If you do not wish to receive commercial and/or marketing email messages
from Fujitsu Australia Software Technology Pty Ltd, please email
unsubscribe(a)fast.au.fujitsu.com


_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev

Hi Tony,

All options are possible. You can run:

- one instance in a one-zone deployment
- one instance in one zone of a multi-zone deployment
- three instances in a one-zone deployment, will all three instances in the
same zone
- three instances in a multi-zone deployment, spreading out the instances
across zones
- three instances in a multi-zone deployment, where you still cram all
instances into one of the many zones

Running multiple instances improves availability in case one node dies, is
temporarily out of commission, being updated, etc. Multiple instances
incurs a performance cost though, since the data needs to be replicated
across nodes in the etcd cluster.

Running across multiple zones further improves availability and
fault-tolerance, since you're now resilient to even a whole zone (e.g.
hardware cluster) going down. This again incurs an additional performance
penalty, as network traffic between zones often suffers higher latency, as
well as a higher risk for network partitions.

I'm not sure where you found "Should be run as 1 node (but can NOT be run as
3 nodes) in environment with only one zone." I do not see it in the release
notes, but perhaps they were edited after you posted this question.

Best,
Amit, CF OSS Release Integration PM
Pivotal Software, Inc.



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p753.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi Gwenn,

Can you elaborate? Are you saying you've actually deployed 3 etcd nodes to
1 zone and you're currently experiencing problems? If so, can you give some
details on the problems?

Thanks,
Amit, CF OSS Release Integration PM
Pivotal Software, Inc.



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p754.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi Amit,

Thank you very much for your clear explanation.

I got the release info(I called it "release Note") from CF release 210,
http://bosh.io/releases/github.com/cloudfoundry/cf-release?version=210

In the "NOTES" part, there is "Upgrade etcd server to 2.0.1" where you can
see
"Should be run as 1 node (for small deployments) or 3 nodes spread across
zones (for HA)"


I will do more test about the third one:
- three instances in a one-zone deployment, will all three instances in the
same zone

Thanks.

Tony




--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p755.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi Amit,

Let me explain the error I got in details.

My env info:
CentOS 6.5,
Openstack Icehouse,
Single-AZ
2 hm9000 instances,
3 etcd instances,


Manifest:
- name: etcd_z1
instances: 3
networks:
- name: cf1
static_ips:
- 100.64.1.21
- 100.64.1.22
- 100.64.1.23
persistent_disk: 10024
properties:
metron_agent:
deployment: metron_agent.deployment
zone: z1
networks:
apps: cf1
etcd:
election_timeout_in_milliseconds: 1000
heartbeat_interval_in_milliseconds: 50
log_sync_timeout_in_seconds: 30
resource_pool: medium_z1
templates:
- name: etcd
release: cf
- name: etcd_metrics_server
release: cf
- name: metron_agent
release: cf
update: {}

properties:
etcd:
machines:
- 100.64.1.21
- 100.64.1.22
- 100.64.1.23
etcd_metrics_server:
nats:
machines:
- 100.64.1.11
- 100.64.1.12
...


I cf push dora app with 2 instances
(https://github.com/cloudfoundry/cf-acceptance-tests/tree/master/assets/dora)

And I can always get response from it. (curl dora.runmyapp.io --> "Hi, I'm
Dora")
The app runs well.

Then I "cf app dora" and got
...
requested state: started
instances: ?/2
...

Then I "cf app dora" again after about 1 minute, and got
...
requested state: started
instances: 2/2
...

The instances' number varies between ?/2 and 2/2 after that.

I also wrote a small script to send "cf app dora" every second and check the
instances' number.
if the number changed, then record it.

Wed Jul 15 06:50:57 UTC 2015 instances: ?/2 (32s)

Wed Jul 15 06:51:29 UTC 2015 instances: 2/2 (6s)

Wed Jul 15 06:51:35 UTC 2015 instances: ?/2 (1m30s)

Wed Jul 15 06:53:05 UTC 2015 instances: 2/2 (17s)

Wed Jul 15 06:53:22 UTC 2015 instances: ?/2 (3m40s)

Wed Jul 15 06:57:02 UTC 2015 instances: 2/2 (21s)

Wed Jul 15 06:57:23 UTC 2015 instances: ?/2 (2m4s)

Wed Jul 15 06:59:27 UTC 2015 instances: 2/2
...


From above we can see that:
1. instance number varies between ?/2 and 2/2
2. "?/2" can be got more often than "2/2"


The app instances' number is always "2/2" when there is only one etcd
instance.
So I reckon the problem is in multi etcd instances.


Other things I tried, but none of them works:

1. Stop etcd service on one etcd vm(monit stop etcd).

2. restart 3 etcd services one by one

3. restart all 3 etcd vms(terminate vms and let them restart
automatically.)

4. Restart two hm9000 vms

5. Restart haproxy(because I don’t know whether the “for HA” means haproxy)
http://bosh.io/releases/github.com/cloudfoundry/cf-release?version=210
Upgrade etcd server to 2.0.1 details
Should be run as 1 node (for small deployments) or 3 nodes spread across
zones (for HA)

6. Add these properties according to
http://bosh.io/jobs/etcd?source=github.com/cloudfoundry/cf-release&version=210
election_timeout_in_milliseconds: 1000
heartbeat_interval_in_milliseconds: 50
log_sync_timeout_in_seconds: 30


Anyway, it doesn't work when "- three instances in a one-zone deployment,
will all three instances in the same zone " as you mentioned.

Do you have any suggestion about it? Or is there any mistake in my manifest?

Thanks,
Tony



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p756.html
Sent from the CF Dev mailing list archive at Nabble.com.

cf project does not support or test cf-release with centos6.5, only ubuntu
14.04.

etcd nodes should not necessarily be aware of which AZs they are in. the
only difference might be in the bosh manifests if they are in different
zones they likely have different job names and you'd need to ensure that
despite the different job names that they were configured to find each
other correctly.

--
Thank you,

James Bayer

Hi James,

Thank you for reply.

I think I didn't describe it clearly. Sorry about that.

The CentOS 6.5 is the OS underlying our OpenStack.

And we are using Ubuntu Trusty stemcell.
(bosh-openstack-kvm-ubuntu-trusty-go_agent)

Regards,
Tony



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p760.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi Tony,

Are you still having this issue?

Having 3 etcd nodes in a single zone should work perfectly fine. Can you
include logs from your HM9k nodes and your etcd nodes covering a few minutes
where the messed up instance reporting issue is happening. Please let me
know if you need guidance in getting logs of a BOSH VM.

Amit



-----
Amit, CF OSS Release Integration PM
Pivotal Software, Inc.
--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p786.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi Amit,

Thanks for your reply.
Yes, we still get stuck here.

And let's me introduce George who is checking this issue in detail. He will
send log and report with more details to you soon.

BTW, We know how to get logs from bosh vms. Thank you very much.

Regards,
Tony



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p788.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi Amit,

It’s me again.

Could you please see log files in the attachment?

We got these errors:

Etcd:
./etcd/etcd.stderr.log:
etcdhttp: unexpected error: etcdserver: request timed out
(but it might be not a bug according to https://github.com/coreos/etcd/issues/3133)

Hm9000
./hm9000/hm9000_analyzer.log , hm9000_apiserver.log and hm9000_sender.log
"error","message":"Store is not fresh - Error:Actual state is not fresh"


Today we tried to run it with 2 etcd instances but not 3, to detect the problem more clearly.

Also, we tried to change properties of etcd:

properties:
…
etcd:
election_timeout_in_milliseconds: 3000 (also tried 5000)
heartbeat_interval_in_milliseconds: 50 (also tried 500, 1000)
log_sync_timeout_in_seconds: 30

It seems that heartbeat_interval_in_milliseconds works
(got ?/2 about 4 times as many as 2/2, when it is 50, and
Got ?/2 about 1.7 times as many as 2/2, when it is 1000).

We may try 2000 or 3000 tomorrow though we think it is too big compared with the default value 50.

Do you have any idea about it?

Thanks,

Tony Li

From: Tony [via CF Dev] [mailto:ml-node+s70369n788h4(a)n6.nabble.com]
Sent: Tuesday, 21 July 2015 9:54 AM
To: Li, Tony
Subject: Re: [cf-dev] 3 etcd nodes don't work well in single zone

Hi Amit,

Thanks for your reply.
Yes, we still get stuck here.

And let's me introduce George who is checking this issue in detail. He will send log and report with more details to you soon.

BTW, We know how to get logs from bosh vms. Thank you very much.

Regards,
Tony
________________________________
If you reply to this email, your message will be added to the discussion below:
http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p788.html
To unsubscribe from [cf-dev] 3 etcd nodes don't work well in single zone, click here<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=746&code=VG9ueWxAZmFzdC5hdS5mdWppdHN1LmNvbXw3NDZ8LTQ5MjU5Njk1Nw==>.
NAML<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.


Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.


If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe(a)fast.au.fujitsu.com


logs.zip (47K) <http://cf-dev.70369.x6.nabble.com/attachment/797/0/logs.zip>




--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p797.html
Sent from the CF Dev mailing list archive at Nabble.com.

You should definitely not run etcd with 2 instances. You can read more about
recommended cluster sizes in the etcd docs:

https://github.com/coreos/etcd/blob/740187f199a12652ca1b7bddb7b3489160103d84/Documentation/admin_guide.md#fault-tolerance-table

I will look at the attached logs and get back to you, but wanted to make
sure to advise you to run either 1 or 3 nodes. With 2, you can wedge the
system, because it will need all nodes to be up to achieve quorum. If you
roll one of the two nodes, it will not be able to rejoin the cluster, and
the service will be stuck in an unavailable state.



-----
Amit, CF OSS Release Integration PM
Pivotal Software, Inc.
--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p809.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi Tony,

The logs you've retrieved only go back to Jul 21, which I can't correlate
with the "?/2" issues you were seeing. If you could possibly record again
a bunch of occurrences of flapping between "2/2" and "?/2" for an app
(along with datetime stamps), and then immediately get logs from *all* the
HM and etcd nodes (`bosh logs` only gets logs from one node at a time), I
can try to dig in more. It's important to get the logs from the HM and
etcd VMs soon after recording the "?/2" events, otherwise BOSH may
rotate/archive the logs and then make them harder to obtain.

Best,
Amit

Hi Amit,

Here is the latest logs I got from etcd and hm9k (I use scp instead of bosh logs to avoid missing something) immediately after finishing test.

May I mention that there is a test folder in the zip file:

test-etcd.sh is a simple script I use, it sends cf app dora every second, and records responses in status.log,

if instance number changed ,then records it in variation.log

In in variation.log, you can see the instance number varies between 2/2 and ?/2 eight times within about 10 minutes.

Thu Jul 23 08:39:29 UTC 2015
instances: 2/2
Thu Jul 23 08:42:59 UTC 2015
instances: ?/2
Thu Jul 23 08:43:36 UTC 2015
instances: 2/2
Thu Jul 23 08:44:55 UTC 2015
instances: ?/2
Thu Jul 23 08:45:32 UTC 2015
instances: 2/2
Thu Jul 23 08:48:31 UTC 2015
instances: ?/2
Thu Jul 23 08:49:02 UTC 2015
instances: 2/2
Thu Jul 23 08:50:05 UTC 2015
instances: ?/2
Thu Jul 23 08:50:41 UTC 2015
instances: 2/2


The start time of this test is “Thu Jul 23 08:39:29 UTC 2015” , it is around "timestamp":1437640773, so I delete most of content before 143763… to make the logs clear.

I didn’t delete any log after 1437640773. If you see the last line of some file(e.g. hm9000_sender.log) is before 1437640773, that just means it didn’t print any log since then.


And I find that at the moments it varies, there isn’t any error recorded in etcd log.

So it seems that the problem is in hm. I’m not sure.

Regards,
Tony

From: Amit Gupta [via CF Dev] [mailto:ml-node+s70369n810h86(a)n6.nabble.com]
Sent: Wednesday, 22 July 2015 10:09 AM
To: Li, Tony
Subject: Re: [cf-dev] 3 etcd nodes don't work well in single zone

Hi Tony,

The logs you've retrieved only go back to Jul 21, which I can't correlate with the "?/2" issues you were seeing. If you could possibly record again a bunch of occurrences of flapping between "2/2" and "?/2" for an app (along with datetime stamps), and then immediately get logs from *all* the HM and etcd nodes (`bosh logs` only gets logs from one node at a time), I can try to dig in more. It's important to get the logs from the HM and etcd VMs soon after recording the "?/2" events, otherwise BOSH may rotate/archive the logs and then make them harder to obtain.

Best,
Amit

On Tue, Jul 21, 2015 at 4:53 PM, Amit Gupta <[hidden email]</user/SendEmail.jtp?type=node&node=810&i=0>> wrote:
You should definitely not run etcd with 2 instances. You can read more about
recommended cluster sizes in the etcd docs:

https://github.com/coreos/etcd/blob/740187f199a12652ca1b7bddb7b3489160103d84/Documentation/admin_guide.md#fault-tolerance-table

I will look at the attached logs and get back to you, but wanted to make
sure to advise you to run either 1 or 3 nodes. With 2, you can wedge the
system, because it will need all nodes to be up to achieve quorum. If you
roll one of the two nodes, it will not be able to rejoin the cluster, and
the service will be stuck in an unavailable state.



-----
Amit, CF OSS Release Integration PM
Pivotal Software, Inc.
--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p809.html
Sent from the CF Dev mailing list archive at Nabble.com.
_______________________________________________
cf-dev mailing list
[hidden email]</user/SendEmail.jtp?type=node&node=810&i=1>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev


_______________________________________________
cf-dev mailing list
[hidden email]</user/SendEmail.jtp?type=node&node=810&i=2>
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
Amit, CF OSS Release Integration PM
Pivotal Software, Inc.

________________________________
If you reply to this email, your message will be added to the discussion below:
http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p810.html
To unsubscribe from [cf-dev] 3 etcd nodes don't work well in single zone, click here<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=746&code=VG9ueWxAZmFzdC5hdS5mdWppdHN1LmNvbXw3NDZ8LTQ5MjU5Njk1Nw==>.
NAML<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>
Disclaimer

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.


Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.


If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe(a)fast.au.fujitsu.com


logs.zip (103K) <http://cf-dev.70369.x6.nabble.com/attachment/847/0/logs.zip>




--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p847.html
Sent from the CF Dev mailing list archive at Nabble.com.

Looks like there are a lot of log lines in the hm.listener component where
it takes to long to save state to etcd. It updates state based on
heartbeats from the DEAs. When the etcd request takes too long, the
listener doesn't mark the data as fresh (it lets a key with a TTL expire).
Then when another component tries to get the state of actual running
instances (this value populates the number of running instances you see
changing in the CLI output), it bails early because it detects the data is
stale. CC can't determine the number of running instances, so it reports
-1 as a sentinel to indicate unknown, which the CLI renders as ?.

The question is why are saves sometimes taking too long, causing the data
to be marked stale so frequently?

Hey Tony,

Any update on your situation?

Amit



-----
Amit, CF OSS Release Integration PM
Pivotal Software, Inc.
--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p900.html
Sent from the CF Dev mailing list archive at Nabble.com.

Hi Amit,

The problem is still there.

We've decided to test it in other env besides openstack.

Thanks for following and we will definitely let you know as soon as we get
new test report.

Regards,
Tony



--
View this message in context: http://cf-dev.70369.x6.nabble.com/cf-dev-3-etcd-nodes-don-t-work-well-in-single-zone-tp746p902.html
Sent from the CF Dev mailing list archive at Nabble.com.