zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Camille Fournier <cami...@apache.org>
Subject Re: ZK read-only issue
Date Fri, 29 Jul 2016 19:07:54 GMT
Ok yeah I think this is reproducible and a bug in the client connection
read-only logic.

On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <camille@apache.org>
wrote:

> I'm looking at the readonly mode code right now and it appears that the
> only way to set readonly mode is a global system property which means that
> the tests for this are only testing across 3 servers, all of which have
> readonly mode set. So, this MAY be a bug, but what a pain to figure out how
> to reproduce.
>
> Global system properties: not even once.
>
> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fpj@apache.org> wrote:
>
>> Not that I'm aware of...
>>
>> -Flavio
>>
>> > On 29 Jul 2016, at 17:24, Benjamin Jaton <benjamin.jaton@gmail.com>
>> wrote:
>> >
>> > Hi Flavio,
>> >
>> > The bug / patch is only for the C client, does this issue affect the
>> java
>> > client as well?
>> >
>> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
>> benjamin.jaton@gmail.com>
>> > wrote:
>> >
>> >> I will definitely try and I'll post an update, thanks again Flavio.
>> >>
>> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fpj@apache.org>
>> wrote:
>> >>
>> >>> I think you're hitting this:
>> >>>
>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>> >>>
>> >>> which is resolved for 3.5.3 and trunk. You can try out the patch.
>> >>>
>> >>> -Flavio
>> >>>
>> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <benjamin.jaton@gmail.com>
>> >>> wrote:
>> >>>>
>> >>>> OK so I did another fresh install this time using 3.5.2-alpha, and
>> the
>> >>> read
>> >>>> only client doesn't seem to try all the hosts available.
>> >>>>
>> >>>> I create my client with:
>> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
>> >>>> 10.11.12.210:2181", 45000, this, true);
>> >>>>
>> >>>> In the following output, it only checks for 10.11.12.4:2181 and
>> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where
>> >>> the
>> >>>> running server is.
>> >>>>
>> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
>> 07/21/2016
>> >>>> 16:24 GMT
>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>> >>> host.name
>> >>>> =w-rli09-ben
>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:java.version=1.8.0_11
>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:java.vendor=Oracle Corporation
>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>>
>> >>>
>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>>
>> >>>
>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:java.io.tmpdir=/tmp
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:java.compiler=<NA>
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>> >>> os.name
>> >>>> =Linux
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:os.arch=amd64
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:os.version=3.8.0-44-generic
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>> >>> user.name
>> >>>> =benji
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:user.home=/home/benji
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:os.memory.free=376MB
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:os.memory.max=5358MB
>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>> >>>> environment:os.memory.total=378MB
>> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper  - Initiating client
>> >>>> connection, connectString=10.11.9.70:2181
>> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181
>> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181
>> >
>> >>>> sessionTimeout=45000
>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
>> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket  -
>> jute.maxbuffer
>> >>> is
>> >>>> 4194304
>> >>>> Successfully established the connection with ZooKeeper
>> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)]
INFO
>> >>>> org.apache.zookeeper.ClientCnxn  - Opening socket connection to
>> server
>> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>.
Will
>> >>> not
>> >>>> attempt to authenticate using SASL (unknown error)*
>> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
>> >>> org.apache.zookeeper.ClientCnxn
>> >>>> - Session 0x0 for server null, unexpected error, closing socket
>> >>> connection
>> >>>> and attempting reconnect
>> >>>> java.net.ConnectException: Connection refused
>> >>>>   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> >>>>   at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>> >>>>   at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>> >>>>   at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
>> >>>> shutdown input
>> >>>> java.nio.channels.ClosedChannelException
>> >>>>   at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>> >>>>   at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>> >>>>   at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>> >>>>   at
>> >>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> >>>>   at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
>> >>>> shutdown output
>> >>>> java.nio.channels.ClosedChannelException
>> >>>>   at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>> >>>>   at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>> >>>>   at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>> >>>>   at
>> >>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> >>>>   at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)]
>> >>> INFO
>> >>>> org.apache.zookeeper.ClientCnxn  - Opening socket connection to
>> server
>> >>>> 10.11.12.210/10.11.12.210:2181 <
>> http://10.11.12.210/10.11.12.210:2181>.
>> >>>> Will not attempt to authenticate using SASL (unknown error)*
>> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
>> >>>> org.apache.zookeeper.ClientCnxn  - Session 0x0 for server null,
>> >>> unexpected
>> >>>> error, closing socket connection and attempting reconnect
>> >>>> java.net.ConnectException: Connection refused
>> >>>>   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>> >>>>   at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>> >>>>   at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>> >>>>   at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
>> >>>> shutdown input
>> >>>> java.nio.channels.ClosedChannelException
>> >>>>   at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>> >>>>   at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>> >>>>   at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>> >>>>   at
>> >>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> >>>>   at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
>> >>>> shutdown output
>> >>>> java.nio.channels.ClosedChannelException
>> >>>>   at
>> >>>>
>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>> >>>>   at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>> >>>>   at
>> >>>>
>> >>>
>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>> >>>>   at
>> >>>>
>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>> >>>>   at
>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>> >>>> *Exception in thread "main"
>> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >>>> KeeperErrorCode = ConnectionLoss for /*
>> >>>>   at
>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> >>>>   at
>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> >>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
>> >>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
>> >>>>   at
>> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
>> >>>>   at
>> >>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>> >>>>
>> >>>>
>> >>>> Does anybody else have issues with the readonly mode?
>> >>>>
>> >>>> Thanks
>> >>>> Benjamin
>> >>>>
>> >>>>
>> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
>> >>> benjamin.jaton@gmail.com>
>> >>>> wrote:
>> >>>>
>> >>>>> Very interesting, thank you Flavio.
>> >>>>>
>> >>>>> That should not happen because I have actually never started
(even
>> >>> never
>> >>>>> created) the other servers. This server is the first one the
client
>> >>> could
>> >>>>> have possibly ever seen. Maybe the server has to have been in
a
>> >>> functioning
>> >>>>> ensemble at least once before being able to be read only able?
>> >>>>> I will recheck all my setup and do it again.
>> >>>>> Thanks again,
>> >>>>> Ben
>> >>>>>
>> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fpj@apache.org>
>> >>> wrote:
>> >>>>>
>> >>>>>> From the logs, it looks like the server the client is trying
to
>> >>> connect
>> >>>>>> to is behind compared to last zxid the client has seen:
>> >>>>>>
>> >>>>>>       Refusing session request for client /10.11.12.4:49237
as it
>> >>> has
>> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another
server
>> >>>>>>
>> >>>>>> I believe the behavior we currently have is that a client
can
>> connect
>> >>> to
>> >>>>>> an RO server, but the server must have a view at least as
recent as
>> >>> the
>> >>>>>> server.
>> >>>>>>
>> >>>>>> -Flavio
>> >>>>>>
>> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
>> benjamin.jaton@gmail.com>
>> >>>>>> wrote:
>> >>>>>>>
>> >>>>>>> Hello,
>> >>>>>>>
>> >>>>>>> I'm having problems using ZK in read-only mode.
>> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I
configure the
>> >>>>>> installation to be in a 3 nodes ensemble.
>> >>>>>>> I start this node only (QA-E8WIN11) with
>> -Dreadonlymode.enabled=true.
>> >>>>>>>
>> >>>>>>> I see that the node QA-E8WIN11 starts the RO server
properly:
>> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server
started
>> >>>>>>>
>> >>>>>>> Then I make a read-only enabled client:
>> >>>>>>>
>> >>>>>>>       ZooKeeper zk = new
>> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
45000,
>> >>> this,
>> >>>>>> true);
>> >>>>>>>       System.out.println("Successfully established the
connection
>> >>>>>> with ZooKeeper");
>> >>>>>>>
>> >>>>>>>       zk.getData("/", false, null);
>> >>>>>>>       System.out.println("Done.");
>> >>>>>>>
>> >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are
not.
>> >>>>>>>
>> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
>> >>>>>>> But when it picks a different node, it fails with:
>> >>>>>>>
>> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating
>> client
>> >>>>>> connection,
>> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>> >>>>>>> Successfully established the connection with ZooKeeper
>> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 -
Opening socket
>> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate
using
>> >>> SASL
>> >>>>>> (unknown error)
>> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 -
Client session
>> >>>>>> timed out, have not heard from server in 15001ms for sessionid
0x0,
>> >>> closing
>> >>>>>> socket connection and attempting reconnect
>> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 -
Opening socket
>> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate
using
>> >>> SASL
>> >>>>>> (unknown error)
>> >>>>>>> Exception in thread "main"
>> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>> >>>>>> KeeperErrorCode = ConnectionLoss for /
>> >>>>>>>   at
>> >>>>>>
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>> >>>>>>>   at
>> >>>>>>
>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> >>>>>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>> >>>>>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>> >>>>>>>   at TestZkReadOnly.run(TestZkReadOnly.java:26)
>> >>>>>>>   at TestZkReadOnly.main(TestZkReadOnly.java:16)
>> >>>>>>>
>> >>>>>>> I'm attaching all the logs. The server and the client
are on the
>> same
>> >>>>>> machine QA-E8WIN11.
>> >>>>>>>
>> >>>>>>> Any help would be greatly appreciated!
>> >>>>>>>
>> >>>>>>> Thanks,
>> >>>>>>> Benjamin Jaton
>> >>>>>>> <zkclient.log><zkserver.log>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>
>> >>>
>> >>
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message