Re: 3 etcd nodes don't work well in single zone


Hi Amit,

It’s me again.

Could you please see log files in the attachment?

We got these errors:

etcdhttp: unexpected error: etcdserver: request timed out
(but it might be not a bug according to

./hm9000/hm9000_analyzer.log , hm9000_apiserver.log and hm9000_sender.log
"error","message":"Store is not fresh - Error:Actual state is not fresh"

Today we tried to run it with 2 etcd instances but not 3, to detect the problem more clearly.

Also, we tried to change properties of etcd:


election_timeout_in_milliseconds: 3000 (also tried 5000)
heartbeat_interval_in_milliseconds: 50 (also tried 500, 1000)
log_sync_timeout_in_seconds: 30

It seems that heartbeat_interval_in_milliseconds works
(got ?/2 about 4 times as many as 2/2, when it is 50, and
Got ?/2 about 1.7 times as many as 2/2, when it is 1000).

We may try 2000 or 3000 tomorrow though we think it is too big compared with the default value 50.

Do you have any idea about it?


Tony Li

From: Tony [via CF Dev] [mailto:ml-node+s70369n788h4(a)]
Sent: Tuesday, 21 July 2015 9:54 AM
To: Li, Tony
Subject: Re: [cf-dev] 3 etcd nodes don't work well in single zone

Hi Amit,

Thanks for your reply.
Yes, we still get stuck here.

And let's me introduce George who is checking this issue in detail. He will send log and report with more details to you soon.

BTW, We know how to get logs from bosh vms. Thank you very much.

If you reply to this email, your message will be added to the discussion below:
To unsubscribe from [cf-dev] 3 etcd nodes don't work well in single zone, click here<>.

The information in this e-mail is confidential and may contain content that is subject to copyright and/or is commercial-in-confidence and is intended only for the use of the above named addressee. If you are not the intended recipient, you are hereby notified that dissemination, copying or use of the information is strictly prohibited. If you have received this e-mail in error, please telephone Fujitsu Australia Software Technology Pty Ltd on + 61 2 9452 9000 or by reply e-mail to the sender and delete the document and all copies thereof.

Whereas Fujitsu Australia Software Technology Pty Ltd would not knowingly transmit a virus within an email communication, it is the receiver’s responsibility to scan all communication and any files attached for computer viruses and other defects. Fujitsu Australia Software Technology Pty Ltd does not accept liability for any loss or damage (whether direct, indirect, consequential or economic) however caused, and whether by negligence or otherwise, which may result directly or indirectly from this communication or any files attached.

If you do not wish to receive commercial and/or marketing email messages from Fujitsu Australia Software Technology Pty Ltd, please email unsubscribe(a) (47K) <>

View this message in context:
Sent from the CF Dev mailing list archive at

Join to automatically receive all group messages.