zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Camille Fournier <cami...@apache.org>
Subject Re: ZK read-only issue
Date Fri, 29 Jul 2016 21:26:38 GMT
Sorry again I'm having reading issues ignore that comment as I see flavio
already answered it

On Jul 29, 2016 5:25 PM, "Camille Fournier" <camille@apache.org> wrote:

> Update,  I was confused by races of my own doing. Was this client
> previously connected when it failed the read only check? The server log is
> rejecting it because the client is ahead in zxid in the first set of logs.
> The other two servers are totally down and it's a new client when you see
> this issue?
>
> On Jul 29, 2016 3:07 PM, "Camille Fournier" <camille@apache.org> wrote:
>
>> Ok yeah I think this is reproducible and a bug in the client connection
>> read-only logic.
>>
>> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <camille@apache.org>
>> wrote:
>>
>>> I'm looking at the readonly mode code right now and it appears that the
>>> only way to set readonly mode is a global system property which means that
>>> the tests for this are only testing across 3 servers, all of which have
>>> readonly mode set. So, this MAY be a bug, but what a pain to figure out how
>>> to reproduce.
>>>
>>> Global system properties: not even once.
>>>
>>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fpj@apache.org>
>>> wrote:
>>>
>>>> Not that I'm aware of...
>>>>
>>>> -Flavio
>>>>
>>>> > On 29 Jul 2016, at 17:24, Benjamin Jaton <benjamin.jaton@gmail.com>
>>>> wrote:
>>>> >
>>>> > Hi Flavio,
>>>> >
>>>> > The bug / patch is only for the C client, does this issue affect the
>>>> java
>>>> > client as well?
>>>> >
>>>> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
>>>> benjamin.jaton@gmail.com>
>>>> > wrote:
>>>> >
>>>> >> I will definitely try and I'll post an update, thanks again Flavio.
>>>> >>
>>>> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fpj@apache.org>
>>>> wrote:
>>>> >>
>>>> >>> I think you're hitting this:
>>>> >>>
>>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
>>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>>>> >>>
>>>> >>> which is resolved for 3.5.3 and trunk. You can try out the patch.
>>>> >>>
>>>> >>> -Flavio
>>>> >>>
>>>> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <benjamin.jaton@gmail.com
>>>> >
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> OK so I did another fresh install this time using 3.5.2-alpha,
and
>>>> the
>>>> >>> read
>>>> >>>> only client doesn't seem to try all the hosts available.
>>>> >>>>
>>>> >>>> I create my client with:
>>>> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
>>>> >>>> 10.11.12.210:2181", 45000, this, true);
>>>> >>>>
>>>> >>>> In the following output, it only checks for 10.11.12.4:2181
and
>>>> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which
is
>>>> where
>>>> >>> the
>>>> >>>> running server is.
>>>> >>>>
>>>> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built
on
>>>> 07/21/2016
>>>> >>>> 16:24 GMT
>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>>>> >>> host.name
>>>> >>>> =w-rli09-ben
>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:java.version=1.8.0_11
>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:java.vendor=Oracle Corporation
>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
>>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>>
>>>> >>>
>>>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>>
>>>> >>>
>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:java.io.tmpdir=/tmp
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:java.compiler=<NA>
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>>>> >>> os.name
>>>> >>>> =Linux
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:os.arch=amd64
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:os.version=3.8.0-44-generic
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>>>> >>> user.name
>>>> >>>> =benji
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:user.home=/home/benji
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:os.memory.free=376MB
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:os.memory.max=5358MB
>>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> >>>> environment:os.memory.total=378MB
>>>> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper  - Initiating
client
>>>> >>>> connection, connectString=10.11.9.70:2181
>>>> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181
>>>> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <
>>>> http://10.11.12.210:2181>
>>>> >>>> sessionTimeout=45000
>>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
>>>> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket  -
>>>> jute.maxbuffer
>>>> >>> is
>>>> >>>> 4194304
>>>> >>>> Successfully established the connection with ZooKeeper
>>>> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)]
>>>> INFO
>>>> >>>> org.apache.zookeeper.ClientCnxn  - Opening socket connection
to
>>>> server
>>>> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>.
>>>> Will
>>>> >>> not
>>>> >>>> attempt to authenticate using SASL (unknown error)*
>>>> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
>>>> >>> org.apache.zookeeper.ClientCnxn
>>>> >>>> - Session 0x0 for server null, unexpected error, closing
socket
>>>> >>> connection
>>>> >>>> and attempting reconnect
>>>> >>>> java.net.ConnectException: Connection refused
>>>> >>>>   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> >>>>   at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>> >>>>   at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>> >>>>   at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception
>>>> during
>>>> >>>> shutdown input
>>>> >>>> java.nio.channels.ClosedChannelException
>>>> >>>>   at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>> >>>>   at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>> >>>>   at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>> >>>>   at
>>>> >>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> >>>>   at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception
>>>> during
>>>> >>>> shutdown output
>>>> >>>> java.nio.channels.ClosedChannelException
>>>> >>>>   at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>> >>>>   at
>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>> >>>>   at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>> >>>>   at
>>>> >>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> >>>>   at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181
>>>> >)]
>>>> >>> INFO
>>>> >>>> org.apache.zookeeper.ClientCnxn  - Opening socket connection
to
>>>> server
>>>> >>>> 10.11.12.210/10.11.12.210:2181 <
>>>> http://10.11.12.210/10.11.12.210:2181>.
>>>> >>>> Will not attempt to authenticate using SASL (unknown error)*
>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
>>>> >>>> org.apache.zookeeper.ClientCnxn  - Session 0x0 for server
null,
>>>> >>> unexpected
>>>> >>>> error, closing socket connection and attempting reconnect
>>>> >>>> java.net.ConnectException: Connection refused
>>>> >>>>   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>> >>>>   at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>> >>>>   at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>> >>>>   at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception
>>>> during
>>>> >>>> shutdown input
>>>> >>>> java.nio.channels.ClosedChannelException
>>>> >>>>   at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>> >>>>   at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>> >>>>   at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>> >>>>   at
>>>> >>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> >>>>   at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception
>>>> during
>>>> >>>> shutdown output
>>>> >>>> java.nio.channels.ClosedChannelException
>>>> >>>>   at
>>>> >>>>
>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>> >>>>   at
>>>> sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>> >>>>   at
>>>> >>>>
>>>> >>>
>>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>> >>>>   at
>>>> >>>>
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>> >>>>   at
>>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> >>>> *Exception in thread "main"
>>>> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> >>>> KeeperErrorCode = ConnectionLoss for /*
>>>> >>>>   at
>>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>> >>>>   at
>>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>> >>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
>>>> >>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
>>>> >>>>   at
>>>> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
>>>> >>>>   at
>>>> >>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>>>> >>>>
>>>> >>>>
>>>> >>>> Does anybody else have issues with the readonly mode?
>>>> >>>>
>>>> >>>> Thanks
>>>> >>>> Benjamin
>>>> >>>>
>>>> >>>>
>>>> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
>>>> >>> benjamin.jaton@gmail.com>
>>>> >>>> wrote:
>>>> >>>>
>>>> >>>>> Very interesting, thank you Flavio.
>>>> >>>>>
>>>> >>>>> That should not happen because I have actually never
started (even
>>>> >>> never
>>>> >>>>> created) the other servers. This server is the first
one the
>>>> client
>>>> >>> could
>>>> >>>>> have possibly ever seen. Maybe the server has to have
been in a
>>>> >>> functioning
>>>> >>>>> ensemble at least once before being able to be read
only able?
>>>> >>>>> I will recheck all my setup and do it again.
>>>> >>>>> Thanks again,
>>>> >>>>> Ben
>>>> >>>>>
>>>> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fpj@apache.org
>>>> >
>>>> >>> wrote:
>>>> >>>>>
>>>> >>>>>> From the logs, it looks like the server the client
is trying to
>>>> >>> connect
>>>> >>>>>> to is behind compared to last zxid the client has
seen:
>>>> >>>>>>
>>>> >>>>>>       Refusing session request for client /10.11.12.4:49237
as
>>>> it
>>>> >>> has
>>>> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try
another server
>>>> >>>>>>
>>>> >>>>>> I believe the behavior we currently have is that
a client can
>>>> connect
>>>> >>> to
>>>> >>>>>> an RO server, but the server must have a view at
least as recent
>>>> as
>>>> >>> the
>>>> >>>>>> server.
>>>> >>>>>>
>>>> >>>>>> -Flavio
>>>> >>>>>>
>>>> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
>>>> benjamin.jaton@gmail.com>
>>>> >>>>>> wrote:
>>>> >>>>>>>
>>>> >>>>>>> Hello,
>>>> >>>>>>>
>>>> >>>>>>> I'm having problems using ZK in read-only mode.
>>>> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha
where I configure
>>>> the
>>>> >>>>>> installation to be in a 3 nodes ensemble.
>>>> >>>>>>> I start this node only (QA-E8WIN11) with
>>>> -Dreadonlymode.enabled=true.
>>>> >>>>>>>
>>>> >>>>>>> I see that the node QA-E8WIN11 starts the RO
server properly:
>>>> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>>>> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only
server started
>>>> >>>>>>>
>>>> >>>>>>> Then I make a read-only enabled client:
>>>> >>>>>>>
>>>> >>>>>>>       ZooKeeper zk = new
>>>> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
>>>> 45000,
>>>> >>> this,
>>>> >>>>>> true);
>>>> >>>>>>>       System.out.println("Successfully established
the
>>>> connection
>>>> >>>>>> with ZooKeeper");
>>>> >>>>>>>
>>>> >>>>>>>       zk.getData("/", false, null);
>>>> >>>>>>>       System.out.println("Done.");
>>>> >>>>>>>
>>>> >>>>>>> Note that only QA-E8WIN11 is started, the other
2 are not.
>>>> >>>>>>>
>>>> >>>>>>> When the ZK client picks QA-E8WIN11 first, it
works.
>>>> >>>>>>> But when it picks a different node, it fails
with:
>>>> >>>>>>>
>>>> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716
- Initiating
>>>> client
>>>> >>>>>> connection,
>>>> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>>>> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>>>> >>>>>>> Successfully established the connection with
ZooKeeper
>>>> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138
- Opening
>>>> socket
>>>> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181
<
>>>> >>>>>> http://10.11.8.230:2181/>. Will not attempt to
authenticate
>>>> using
>>>> >>> SASL
>>>> >>>>>> (unknown error)
>>>> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251
- Client
>>>> session
>>>> >>>>>> timed out, have not heard from server in 15001ms
for sessionid
>>>> 0x0,
>>>> >>> closing
>>>> >>>>>> socket connection and attempting reconnect
>>>> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138
- Opening
>>>> socket
>>>> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181
<
>>>> >>>>>> http://10.11.8.232:2181/>. Will not attempt to
authenticate
>>>> using
>>>> >>> SASL
>>>> >>>>>> (unknown error)
>>>> >>>>>>> Exception in thread "main"
>>>> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> >>>>>> KeeperErrorCode = ConnectionLoss for /
>>>> >>>>>>>   at
>>>> >>>>>>
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>> >>>>>>>   at
>>>> >>>>>>
>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>> >>>>>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>>>> >>>>>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>>>> >>>>>>>   at TestZkReadOnly.run(TestZkReadOnly.java:26)
>>>> >>>>>>>   at TestZkReadOnly.main(TestZkReadOnly.java:16)
>>>> >>>>>>>
>>>> >>>>>>> I'm attaching all the logs. The server and the
client are on
>>>> the same
>>>> >>>>>> machine QA-E8WIN11.
>>>> >>>>>>>
>>>> >>>>>>> Any help would be greatly appreciated!
>>>> >>>>>>>
>>>> >>>>>>> Thanks,
>>>> >>>>>>> Benjamin Jaton
>>>> >>>>>>> <zkclient.log><zkserver.log>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>
>>>> >>>
>>>> >>>
>>>> >>
>>>>
>>>>
>>>
>>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message