Issue with etcd since v211


Scott Taggart <staggart@...>
 

Hi folks,



Just upgraded from v210 to v211 and seem to have failures within etcd.
they won't start



On the etc.d box (10.2.0.40 in this instance, I also have one on
10.3.0.40), I get.



Under etcd_ctl.err.log:

[2015-06-10 16:29:51+0000] + /var/vcap/packages/etcd/etcdctl -C
http://10.2.0.40:4001 ls

[2015-06-10 16:29:51+0000] Error: cannot sync with the cluster using
endpoints http://10.2.0.40:4001

[2015-06-10 16:29:51+0000] + sleep 1

[2015-06-10 16:29:52+0000] + false

[2015-06-10 16:29:52+0000] + safe_teardown 'af244208330117aa:
name=etcd_z2-0 peerURLs=http://10.3.0.40:7001
clientURLs=http://10.3.0.40:4001

[2015-06-10 16:29:52+0000] c2e63d26f18f2cd2: name=
peerURLs=http://10.2.0.40:7001 clientURLs='

[2015-06-10 16:29:52+0000] + prior_cluster_had_other_nodes
'af244208330117aa: name=etcd_z2-0 peerURLs=http://10.3.0.40:7001
clientURLs=http://10.3.0.40:4001

[2015-06-10 16:29:52+0000] c2e63d26f18f2cd2: name=
peerURLs=http://10.2.0.40:7001 clientURLs='

[2015-06-10 16:29:52+0000] ++ wc -l

[2015-06-10 16:29:52+0000] + '[' 2 -ne 1 ']'

[2015-06-10 16:29:52+0000] ++ extract_my_id 'af244208330117aa:
name=etcd_z2-0 peerURLs=http://10.3.0.40:7001
clientURLs=http://10.3.0.40:4001

[2015-06-10 16:29:52+0000] c2e63d26f18f2cd2: name=
peerURLs=http://10.2.0.40:7001 clientURLs='

[2015-06-10 16:29:52+0000] ++ sed 's/:.*//'

[2015-06-10 16:29:52+0000] ++ grep 10.2.0.40

[2015-06-10 16:29:52+0000] ++ echo 'af244208330117aa: name=etcd_z2-0
peerURLs=http://10.3.0.40:7001 clientURLs=http://10.3.0.40:4001

[2015-06-10 16:29:52+0000] c2e63d26f18f2cd2: name=
peerURLs=http://10.2.0.40:7001 clientURLs='

[2015-06-10 16:29:52+0000] + my_id=c2e63d26f18f2cd2

[2015-06-10 16:29:52+0000] + '[' '!' -z c2e63d26f18f2cd2 ']'

[2015-06-10 16:29:52+0000] + member_remove c2e63d26f18f2cd2

[2015-06-10 16:29:52+0000] + /var/vcap/packages/etcd/etcdctl -C
http://10.2.0.40:4001,http://10.3.0.40:4001 member remove c2e63d26f18f2cd2



Under etcd.stderr.log:

2015/06/10 16:30:32 etcd: listening for peers on http://0.0.0.0:7001

2015/06/10 16:30:32 etcd: listening for client requests on
http://0.0.0.0:4001

2015/06/10 16:30:32 etcd: stopping listening for client requests on
http://0.0.0.0:4001

2015/06/10 16:30:32 etcd: stopping listening for peers on
http://0.0.0.0:7001

2015/06/10 16:30:32 etcd: error validating peerURLs
=http://10.2.0.40:7001,etcd_z2-0=http://10.3.0.40:7001: member count is
unequal



Any tips at all?



Notice:
This message contains information that may be privileged or confidential and is the property of Skyscape. It is intended only for the person to whom it is addressed. If you are not the intended recipient, you are not authorised to read, print, retain, copy, disseminate, distribute, or use this message or any part thereof. If you receive this message in error, please notify the sender immediately and delete all copies of this message. Skyscape reserves the right to monitor all e-mail communications through its networks. Skyscape Cloud Services Limited is registered in England and Wales: Company No: 07619797. Registered office: Hartham Park, Hartham, Corsham, Wiltshire SN13 0RP.

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________


CF Runtime
 

It looks like you are trying to run with two etcd instances. etcd requires
either one instance, or three or more. You can see details here:
https://github.com/coreos/etcd/blob/master/Documentation/admin_guide.md#fault-tolerance-table

The default cf-release templates normally have two instances in one zone,
and a single instance in the other zone.

CF Runtime Team
Joseph & Marc

On Wed, Jun 10, 2015 at 9:33 AM, Scott Taggart <staggart(a)skyscapecloud.com>
wrote:

Hi folks,



Just upgraded from v210 to v211 and seem to have failures within etcd…
they won’t start



On the etc.d box (10.2.0.40 in this instance, I also have one on
10.3.0.40), I get…



Under etcd_ctl.err.log:

[2015-06-10 16:29:51+0000] + /var/vcap/packages/etcd/etcdctl -C
http://10.2.0.40:4001 ls

[2015-06-10 16:29:51+0000] Error: cannot sync with the cluster using
endpoints http://10.2.0.40:4001

[2015-06-10 16:29:51+0000] + sleep 1

[2015-06-10 16:29:52+0000] + false

[2015-06-10 16:29:52+0000] + safe_teardown 'af244208330117aa:
name=etcd_z2-0 peerURLs=http://10.3.0.40:7001 clientURLs=
http://10.3.0.40:4001

[2015-06-10 16:29:52+0000] c2e63d26f18f2cd2: name= peerURLs=
http://10.2.0.40:7001 clientURLs='

[2015-06-10 16:29:52+0000] + prior_cluster_had_other_nodes
'af244208330117aa: name=etcd_z2-0 peerURLs=http://10.3.0.40:7001
clientURLs=http://10.3.0.40:4001

[2015-06-10 16:29:52+0000] c2e63d26f18f2cd2: name= peerURLs=
http://10.2.0.40:7001 clientURLs='

[2015-06-10 16:29:52+0000] ++ wc -l

[2015-06-10 16:29:52+0000] + '[' 2 -ne 1 ']'

[2015-06-10 16:29:52+0000] ++ extract_my_id 'af244208330117aa:
name=etcd_z2-0 peerURLs=http://10.3.0.40:7001 clientURLs=
http://10.3.0.40:4001

[2015-06-10 16:29:52+0000] c2e63d26f18f2cd2: name= peerURLs=
http://10.2.0.40:7001 clientURLs='

[2015-06-10 16:29:52+0000] ++ sed 's/:.*//'

[2015-06-10 16:29:52+0000] ++ grep 10.2.0.40

[2015-06-10 16:29:52+0000] ++ echo 'af244208330117aa: name=etcd_z2-0
peerURLs=http://10.3.0.40:7001 clientURLs=http://10.3.0.40:4001

[2015-06-10 16:29:52+0000] c2e63d26f18f2cd2: name= peerURLs=
http://10.2.0.40:7001 clientURLs='

[2015-06-10 16:29:52+0000] + my_id=c2e63d26f18f2cd2

[2015-06-10 16:29:52+0000] + '[' '!' -z c2e63d26f18f2cd2 ']'

[2015-06-10 16:29:52+0000] + member_remove c2e63d26f18f2cd2

[2015-06-10 16:29:52+0000] + /var/vcap/packages/etcd/etcdctl -C
http://10.2.0.40:4001,http://10.3.0.40:4001 member remove c2e63d26f18f2cd2



Under etcd.stderr.log:

2015/06/10 16:30:32 etcd: listening for peers on http://0.0.0.0:7001

2015/06/10 16:30:32 etcd: listening for client requests on
http://0.0.0.0:4001

2015/06/10 16:30:32 etcd: stopping listening for client requests on
http://0.0.0.0:4001

2015/06/10 16:30:32 etcd: stopping listening for peers on
http://0.0.0.0:7001

2015/06/10 16:30:32 etcd: error validating peerURLs =http://10.2.0.40:7001
,etcd_z2-0=http://10.3.0.40:7001: member count is unequal



Any tips at all?

Notice:
This message contains information that may be privileged or confidential
and is the property of Skyscape. It is intended only for the person to whom
it is addressed. If you are not the intended recipient, you are not
authorised to read, print, retain, copy, disseminate, distribute, or use
this message or any part thereof. If you receive this message in error,
please notify the sender immediately and delete all copies of this message.
Skyscape reserves the right to monitor all e-mail communications through
its networks. Skyscape Cloud Services Limited is registered in England and
Wales: Company No: 07619797. Registered office: Hartham Park, Hartham,
Corsham, Wiltshire SN13 0RP.

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
______________________________________________________________________

_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev