zookeeper-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthew Hertz (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (ZOOKEEPER-3490) Zookeeper followers not reflecting writes (after months)
Date Fri, 02 Aug 2019 14:53:00 GMT

     [ https://issues.apache.org/jira/browse/ZOOKEEPER-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matthew Hertz updated ZOOKEEPER-3490:
-------------------------------------
    Description: 
Hi,

We have a 3 node Zookeeper cluster. There are a number of znode's on the leader that are not
visible on the followers.
{code:java}
$ zkCli -server <server 1> (follower) 
[zk: <server 1>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
abcd.company.com:<built-in function getpid>
cZxid = 0xf00061d68
ctime = Thu Nov 01 12:40:33 GMT 2018
mZxid = 0xf00061d68
mtime = Thu Nov 01 12:40:33 GMT 2018
pZxid = 0xf00061d68
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x500be5318d60407
dataLength = 58
numChildren = 0
{code}
{code:java}
$ zkCli -server <server 2> (leader)
[zk: <server2>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
Node does not exist: /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
{code}
{code:java}
$ zkCli -server <server 3> (follower)
[zk: <server3>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
abcd.company.com:<built-in function getpid>
cZxid = 0xf00061d68
ctime = Thu Nov 01 12:40:33 GMT 2018
mZxid = 0xf00061d68
mtime = Thu Nov 01 12:40:33 GMT 2018
pZxid = 0xf00061d68
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x500be5318d60407
dataLength = 58
numChildren = 0
{code}
These nodes are ephemeral nodes. The sessions no longer exist. There are 6 znodes in this
'inconsistent' state. The cluster is currently connected - there are no networking partitions
currently.

We're at a loss for how to both debug and fix this. Restarting the Zookeeper followers presumably
will not help? Are all nodes ever force-synced from the leader?

Help would be appreciated. If any more information would be helpful it can be provided, however
we will likely have to resolve this issue one way or another in the near future.

Thanks

 

  was:
Hi,

We have a 3 node Zookeeper cluster. There are a number of znode's on the leader that are not
visible on the followers.
{code:java}
$ zkCli -server <server 1> (follower) 
[zk: <server 1>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
abcd.company.com:<built-in function getpid>
cZxid = 0xf00061d68
ctime = Thu Nov 01 12:40:33 GMT 2018
mZxid = 0xf00061d68
mtime = Thu Nov 01 12:40:33 GMT 2018
pZxid = 0xf00061d68
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x500be5318d60407
dataLength = 58
numChildren = 0
{code}
{code:java}
$ zkCli -server <server 2> (leader)
[zk: <server2>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
Node does not exist: /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
{code}
{code:java}
$ zkCli -server <server 3> (follower)
[zk: <server3>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
abcd.company.com:<built-in function getpid>
cZxid = 0xf00061d68
ctime = Thu Nov 01 12:40:33 GMT 2018
mZxid = 0xf00061d68
mtime = Thu Nov 01 12:40:33 GMT 2018
pZxid = 0xf00061d68
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x500be5318d60407
dataLength = 58
numChildren = 0
{code}
These nodes are ephemeral nodes. The sessions no longer exist. There are 6 znodes in this
'inconsistent' state. The cluster is currently connected - there are no networking partitions
currently.

We're at a loss for how to both debug and fix this. Restarting the Zookeeper followers presumably
will not help? Are all nodes ever force-synced from the leader?

Help would be appreciated.

Thanks

 

        Summary: Zookeeper followers not reflecting writes (after months)  (was: Zookeeper
follower's not reflecting writes (after months))

> Zookeeper followers not reflecting writes (after months)
> --------------------------------------------------------
>
>                 Key: ZOOKEEPER-3490
>                 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3490
>             Project: ZooKeeper
>          Issue Type: Bug
>    Affects Versions: 3.4.13
>            Reporter: Matthew Hertz
>            Priority: Major
>
> Hi,
> We have a 3 node Zookeeper cluster. There are a number of znode's on the leader that
are not visible on the followers.
> {code:java}
> $ zkCli -server <server 1> (follower) 
> [zk: <server 1>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
> abcd.company.com:<built-in function getpid>
> cZxid = 0xf00061d68
> ctime = Thu Nov 01 12:40:33 GMT 2018
> mZxid = 0xf00061d68
> mtime = Thu Nov 01 12:40:33 GMT 2018
> pZxid = 0xf00061d68
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x500be5318d60407
> dataLength = 58
> numChildren = 0
> {code}
> {code:java}
> $ zkCli -server <server 2> (leader)
> [zk: <server2>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
Node does not exist: /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
> {code}
> {code:java}
> $ zkCli -server <server 3> (follower)
> [zk: <server3>(CONNECTED) 0] get /pyMkdProducer/SNAP/lock/c4a62c9fdfdc412fac3818bbb2af3a0f__lock__0000000040
> abcd.company.com:<built-in function getpid>
> cZxid = 0xf00061d68
> ctime = Thu Nov 01 12:40:33 GMT 2018
> mZxid = 0xf00061d68
> mtime = Thu Nov 01 12:40:33 GMT 2018
> pZxid = 0xf00061d68
> cversion = 0
> dataVersion = 0
> aclVersion = 0
> ephemeralOwner = 0x500be5318d60407
> dataLength = 58
> numChildren = 0
> {code}
> These nodes are ephemeral nodes. The sessions no longer exist. There are 6 znodes in
this 'inconsistent' state. The cluster is currently connected - there are no networking partitions
currently.
> We're at a loss for how to both debug and fix this. Restarting the Zookeeper followers
presumably will not help? Are all nodes ever force-synced from the leader?
> Help would be appreciated. If any more information would be helpful it can be provided,
however we will likely have to resolve this issue one way or another in the near future.
> Thanks
>  



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Mime
View raw message