zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <...@apache.org>
Subject Re: ZK read-only issue
Date Fri, 29 Jul 2016 16:27:14 GMT
Not that I'm aware of...

-Flavio

> On 29 Jul 2016, at 17:24, Benjamin Jaton <benjamin.jaton@gmail.com> wrote:
> 
> Hi Flavio,
> 
> The bug / patch is only for the C client, does this issue affect the java
> client as well?
> 
> On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <benjamin.jaton@gmail.com>
> wrote:
> 
>> I will definitely try and I'll post an update, thanks again Flavio.
>> 
>> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fpj@apache.org> wrote:
>> 
>>> I think you're hitting this:
>>> 
>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
>>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>>> 
>>> which is resolved for 3.5.3 and trunk. You can try out the patch.
>>> 
>>> -Flavio
>>> 
>>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <benjamin.jaton@gmail.com>
>>> wrote:
>>>> 
>>>> OK so I did another fresh install this time using 3.5.2-alpha, and the
>>> read
>>>> only client doesn't seem to try all the hosts available.
>>>> 
>>>> I create my client with:
>>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
>>>> 10.11.12.210:2181", 45000, this, true);
>>>> 
>>>> In the following output, it only checks for 10.11.12.4:2181 and
>>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where
>>> the
>>>> running server is.
>>>> 
>>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on 07/21/2016
>>>> 16:24 GMT
>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>>> host.name
>>>> =w-rli09-ben
>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:java.version=1.8.0_11
>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:java.vendor=Oracle Corporation
>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
>>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> 
>>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> 
>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:java.io.tmpdir=/tmp
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:java.compiler=<NA>
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>>> os.name
>>>> =Linux
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:os.arch=amd64
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:os.version=3.8.0-44-generic
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>>> user.name
>>>> =benji
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:user.home=/home/benji
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:os.memory.free=376MB
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:os.memory.max=5358MB
>>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>>> environment:os.memory.total=378MB
>>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper  - Initiating client
>>>> connection, connectString=10.11.9.70:2181
>>>> <http://10.11.9.70:2181>,10.11.12.4:2181
>>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181>
>>>> sessionTimeout=45000 watcher=testzkclient.TestZkReadOnlyClient@5010be6*
>>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket  - jute.maxbuffer
>>> is
>>>> 4194304
>>>> Successfully established the connection with ZooKeeper
>>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] INFO
>>>> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
>>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. Will
>>> not
>>>> attempt to authenticate using SASL (unknown error)*
>>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
>>> org.apache.zookeeper.ClientCnxn
>>>> - Session 0x0 for server null, unexpected error, closing socket
>>> connection
>>>> and attempting reconnect
>>>> java.net.ConnectException: Connection refused
>>>>   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>   at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>>   at
>>>> 
>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>>   at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
>>>> shutdown input
>>>> java.nio.channels.ClosedChannelException
>>>>   at
>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>>   at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>>   at
>>>> 
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>>   at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>   at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
>>>> shutdown output
>>>> java.nio.channels.ClosedChannelException
>>>>   at
>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>>   at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>>   at
>>>> 
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>>   at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>   at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)]
>>> INFO
>>>> org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
>>>> 10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181>.
>>>> Will not attempt to authenticate using SASL (unknown error)*
>>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
>>>> org.apache.zookeeper.ClientCnxn  - Session 0x0 for server null,
>>> unexpected
>>>> error, closing socket connection and attempting reconnect
>>>> java.net.ConnectException: Connection refused
>>>>   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>>>   at
>>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>>>   at
>>>> 
>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>>>   at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
>>>> shutdown input
>>>> java.nio.channels.ClosedChannelException
>>>>   at
>>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>>>   at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>>>   at
>>>> 
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>>>   at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>   at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
>>>> shutdown output
>>>> java.nio.channels.ClosedChannelException
>>>>   at
>>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>>>   at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>>>   at
>>>> 
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>>>   at
>>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>>>   at
>>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>>> *Exception in thread "main"
>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>> KeeperErrorCode = ConnectionLoss for /*
>>>>   at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>>   at
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
>>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
>>>>   at
>>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
>>>>   at
>>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>>>> 
>>>> 
>>>> Does anybody else have issues with the readonly mode?
>>>> 
>>>> Thanks
>>>> Benjamin
>>>> 
>>>> 
>>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
>>> benjamin.jaton@gmail.com>
>>>> wrote:
>>>> 
>>>>> Very interesting, thank you Flavio.
>>>>> 
>>>>> That should not happen because I have actually never started (even
>>> never
>>>>> created) the other servers. This server is the first one the client
>>> could
>>>>> have possibly ever seen. Maybe the server has to have been in a
>>> functioning
>>>>> ensemble at least once before being able to be read only able?
>>>>> I will recheck all my setup and do it again.
>>>>> Thanks again,
>>>>> Ben
>>>>> 
>>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fpj@apache.org>
>>> wrote:
>>>>> 
>>>>>> From the logs, it looks like the server the client is trying to
>>> connect
>>>>>> to is behind compared to last zxid the client has seen:
>>>>>> 
>>>>>>       Refusing session request for client /10.11.12.4:49237 as it
>>> has
>>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another server
>>>>>> 
>>>>>> I believe the behavior we currently have is that a client can connect
>>> to
>>>>>> an RO server, but the server must have a view at least as recent
as
>>> the
>>>>>> server.
>>>>>> 
>>>>>> -Flavio
>>>>>> 
>>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <benjamin.jaton@gmail.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> Hello,
>>>>>>> 
>>>>>>> I'm having problems using ZK in read-only mode.
>>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure
the
>>>>>> installation to be in a 3 nodes ensemble.
>>>>>>> I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=true.
>>>>>>> 
>>>>>>> I see that the node QA-E8WIN11 starts the RO server properly:
>>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
>>>>>>> 
>>>>>>> Then I make a read-only enabled client:
>>>>>>> 
>>>>>>>       ZooKeeper zk = new
>>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000,
>>> this,
>>>>>> true);
>>>>>>>       System.out.println("Successfully established the connection
>>>>>> with ZooKeeper");
>>>>>>> 
>>>>>>>       zk.getData("/", false, null);
>>>>>>>       System.out.println("Done.");
>>>>>>> 
>>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not.
>>>>>>> 
>>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
>>>>>>> But when it picks a different node, it fails with:
>>>>>>> 
>>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
client
>>>>>> connection,
>>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>>>>>>> Successfully established the connection with ZooKeeper
>>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate using
>>> SASL
>>>>>> (unknown error)
>>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session
>>>>>> timed out, have not heard from server in 15001ms for sessionid 0x0,
>>> closing
>>>>>> socket connection and attempting reconnect
>>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
>>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate using
>>> SASL
>>>>>> (unknown error)
>>>>>>> Exception in thread "main"
>>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>>>>> KeeperErrorCode = ConnectionLoss for /
>>>>>>>   at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>>>>>>   at
>>>>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>>>>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>>>>>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>>>>>>>   at TestZkReadOnly.run(TestZkReadOnly.java:26)
>>>>>>>   at TestZkReadOnly.main(TestZkReadOnly.java:16)
>>>>>>> 
>>>>>>> I'm attaching all the logs. The server and the client are on
the same
>>>>>> machine QA-E8WIN11.
>>>>>>> 
>>>>>>> Any help would be greatly appreciated!
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Benjamin Jaton
>>>>>>> <zkclient.log><zkserver.log>
>>>>>> 
>>>>>> 
>>>>> 
>>> 
>>> 
>> 


Mime
View raw message