zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Camille Fournier <cami...@apache.org>
Subject Re: ZK read-only issue
Date Fri, 29 Jul 2016 21:25:28 GMT
Update,  I was confused by races of my own doing. Was this client
previously connected when it failed the read only check? The server log is
rejecting it because the client is ahead in zxid in the first set of logs.
The other two servers are totally down and it's a new client when you see
this issue?

On Jul 29, 2016 3:07 PM, "Camille Fournier" <camille@apache.org> wrote:

> Ok yeah I think this is reproducible and a bug in the client connection
> read-only logic.
>
> On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier <camille@apache.org>
> wrote:
>
>> I'm looking at the readonly mode code right now and it appears that the
>> only way to set readonly mode is a global system property which means that
>> the tests for this are only testing across 3 servers, all of which have
>> readonly mode set. So, this MAY be a bug, but what a pain to figure out how
>> to reproduce.
>>
>> Global system properties: not even once.
>>
>> On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira <fpj@apache.org>
>> wrote:
>>
>>> Not that I'm aware of...
>>>
>>> -Flavio
>>>
>>> > On 29 Jul 2016, at 17:24, Benjamin Jaton <benjamin.jaton@gmail.com>
>>> wrote:
>>> >
>>> > Hi Flavio,
>>> >
>>> > The bug / patch is only for the C client, does this issue affect the
>>> java
>>> > client as well?
>>> >
>>> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton <
>>> benjamin.jaton@gmail.com>
>>> > wrote:
>>> >
>>> >> I will definitely try and I'll post an update, thanks again Flavio.
>>> >>
>>> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fpj@apache.org>
>>> wrote:
>>> >>
>>> >>> I think you're hitting this:
>>> >>>
>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
>>> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>>> >>>
>>> >>> which is resolved for 3.5.3 and trunk. You can try out the patch.
>>> >>>
>>> >>> -Flavio
>>> >>>
>>> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton <benjamin.jaton@gmail.com>
>>> >>> wrote:
>>> >>>>
>>> >>>> OK so I did another fresh install this time using 3.5.2-alpha,
and
>>> the
>>> >>> read
>>> >>>> only client doesn't seem to try all the hosts available.
>>> >>>>
>>> >>>> I create my client with:
>>> >>>> ZooKeeper zk = new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
>>> >>>> 10.11.12.210:2181", 45000, this, true);
>>> >>>>
>>> >>>> In the following output, it only checks for 10.11.12.4:2181
and
>>> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which
is
>>> where
>>> >>> the
>>> >>>> running server is.
>>> >>>>
>>> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:zookeeper.version=3.5.2-alpha-1753710, built on
>>> 07/21/2016
>>> >>>> 16:24 GMT
>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>>> >>> host.name
>>> >>>> =w-rli09-ben
>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:java.version=1.8.0_11
>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:java.vendor=Oracle Corporation
>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:java.home=/usr/local/apps/jdk1.8.0_11/jre
>>> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>>
>>> >>>
>>> environment:java.class.path=/home/benji/workspace-trunk/testzkclient/bin:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapper-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/javacc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alpha.jar
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>>
>>> >>>
>>> environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:java.io.tmpdir=/tmp
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:java.compiler=<NA>
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>>> >>> os.name
>>> >>>> =Linux
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:os.arch=amd64
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:os.version=3.8.0-44-generic
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
>>> >>> user.name
>>> >>>> =benji
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:user.home=/home/benji
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:user.dir=/home/benji/workspace-trunk/testzkclient
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:os.memory.free=376MB
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:os.memory.max=5358MB
>>> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
>>> >>>> environment:os.memory.total=378MB
>>> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper  - Initiating
client
>>> >>>> connection, connectString=10.11.9.70:2181
>>> >>>> <http://10.11.9.70:2181>,10.11.12.4:2181
>>> >>>> <http://10.11.12.4:2181>,10.11.12.210:2181 <
>>> http://10.11.12.210:2181>
>>> >>>> sessionTimeout=45000
>>> watcher=testzkclient.TestZkReadOnlyClient@5010be6*
>>> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket  -
>>> jute.maxbuffer
>>> >>> is
>>> >>>> 4194304
>>> >>>> Successfully established the connection with ZooKeeper
>>> >>>> *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)]
>>> INFO
>>> >>>> org.apache.zookeeper.ClientCnxn  - Opening socket connection
to
>>> server
>>> >>>> 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>.
>>> Will
>>> >>> not
>>> >>>> attempt to authenticate using SASL (unknown error)*
>>> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN
>>> >>> org.apache.zookeeper.ClientCnxn
>>> >>>> - Session 0x0 for server null, unexpected error, closing socket
>>> >>> connection
>>> >>>> and attempting reconnect
>>> >>>> java.net.ConnectException: Connection refused
>>> >>>>   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> >>>>   at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>> >>>>   at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>> >>>>   at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception
>>> during
>>> >>>> shutdown input
>>> >>>> java.nio.channels.ClosedChannelException
>>> >>>>   at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>> >>>>   at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>> >>>>   at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>> >>>>   at
>>> >>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>> >>>>   at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG
>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception
>>> during
>>> >>>> shutdown output
>>> >>>> java.nio.channels.ClosedChannelException
>>> >>>>   at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>> >>>>   at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>> >>>>   at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>> >>>>   at
>>> >>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>> >>>>   at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>> >>>> *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181
>>> >)]
>>> >>> INFO
>>> >>>> org.apache.zookeeper.ClientCnxn  - Opening socket connection
to
>>> server
>>> >>>> 10.11.12.210/10.11.12.210:2181 <
>>> http://10.11.12.210/10.11.12.210:2181>.
>>> >>>> Will not attempt to authenticate using SASL (unknown error)*
>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN
>>> >>>> org.apache.zookeeper.ClientCnxn  - Session 0x0 for server null,
>>> >>> unexpected
>>> >>>> error, closing socket connection and attempting reconnect
>>> >>>> java.net.ConnectException: Connection refused
>>> >>>>   at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
>>> >>>>   at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
>>> >>>>   at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357)
>>> >>>>   at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception
>>> during
>>> >>>> shutdown input
>>> >>>> java.nio.channels.ClosedChannelException
>>> >>>>   at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
>>> >>>>   at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
>>> >>>>   at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:198)
>>> >>>>   at
>>> >>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>> >>>>   at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG
>>> >>>> org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception
>>> during
>>> >>>> shutdown output
>>> >>>> java.nio.channels.ClosedChannelException
>>> >>>>   at
>>> >>>>
>>> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
>>> >>>>   at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
>>> >>>>   at
>>> >>>>
>>> >>>
>>> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:205)
>>> >>>>   at
>>> >>>>
>>> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325)
>>> >>>>   at
>>> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
>>> >>>> *Exception in thread "main"
>>> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> >>>> KeeperErrorCode = ConnectionLoss for /*
>>> >>>>   at
>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>> >>>>   at
>>> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>> >>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
>>> >>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
>>> >>>>   at
>>> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26)
>>> >>>>   at
>>> >>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
>>> >>>>
>>> >>>>
>>> >>>> Does anybody else have issues with the readonly mode?
>>> >>>>
>>> >>>> Thanks
>>> >>>> Benjamin
>>> >>>>
>>> >>>>
>>> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
>>> >>> benjamin.jaton@gmail.com>
>>> >>>> wrote:
>>> >>>>
>>> >>>>> Very interesting, thank you Flavio.
>>> >>>>>
>>> >>>>> That should not happen because I have actually never started
(even
>>> >>> never
>>> >>>>> created) the other servers. This server is the first one
the client
>>> >>> could
>>> >>>>> have possibly ever seen. Maybe the server has to have been
in a
>>> >>> functioning
>>> >>>>> ensemble at least once before being able to be read only
able?
>>> >>>>> I will recheck all my setup and do it again.
>>> >>>>> Thanks again,
>>> >>>>> Ben
>>> >>>>>
>>> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fpj@apache.org>
>>> >>> wrote:
>>> >>>>>
>>> >>>>>> From the logs, it looks like the server the client is
trying to
>>> >>> connect
>>> >>>>>> to is behind compared to last zxid the client has seen:
>>> >>>>>>
>>> >>>>>>       Refusing session request for client /10.11.12.4:49237
as it
>>> >>> has
>>> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another
server
>>> >>>>>>
>>> >>>>>> I believe the behavior we currently have is that a client
can
>>> connect
>>> >>> to
>>> >>>>>> an RO server, but the server must have a view at least
as recent
>>> as
>>> >>> the
>>> >>>>>> server.
>>> >>>>>>
>>> >>>>>> -Flavio
>>> >>>>>>
>>> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <
>>> benjamin.jaton@gmail.com>
>>> >>>>>> wrote:
>>> >>>>>>>
>>> >>>>>>> Hello,
>>> >>>>>>>
>>> >>>>>>> I'm having problems using ZK in read-only mode.
>>> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where
I configure
>>> the
>>> >>>>>> installation to be in a 3 nodes ensemble.
>>> >>>>>>> I start this node only (QA-E8WIN11) with
>>> -Dreadonlymode.enabled=true.
>>> >>>>>>>
>>> >>>>>>> I see that the node QA-E8WIN11 starts the RO server
properly:
>>> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
>>> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server
started
>>> >>>>>>>
>>> >>>>>>> Then I make a read-only enabled client:
>>> >>>>>>>
>>> >>>>>>>       ZooKeeper zk = new
>>> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181",
>>> 45000,
>>> >>> this,
>>> >>>>>> true);
>>> >>>>>>>       System.out.println("Successfully established
the connection
>>> >>>>>> with ZooKeeper");
>>> >>>>>>>
>>> >>>>>>>       zk.getData("/", false, null);
>>> >>>>>>>       System.out.println("Done.");
>>> >>>>>>>
>>> >>>>>>> Note that only QA-E8WIN11 is started, the other
2 are not.
>>> >>>>>>>
>>> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works.
>>> >>>>>>> But when it picks a different node, it fails with:
>>> >>>>>>>
>>> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716
- Initiating
>>> client
>>> >>>>>> connection,
>>> >>> connectString=QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
>>> >>>>>> sessionTimeout=45000 watcher=TestZkReadOnly@68de145
>>> >>>>>>> Successfully established the connection with ZooKeeper
>>> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138
- Opening
>>> socket
>>> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 <
>>> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate
using
>>> >>> SASL
>>> >>>>>> (unknown error)
>>> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251
- Client
>>> session
>>> >>>>>> timed out, have not heard from server in 15001ms for
sessionid
>>> 0x0,
>>> >>> closing
>>> >>>>>> socket connection and attempting reconnect
>>> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138
- Opening
>>> socket
>>> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 <
>>> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate
using
>>> >>> SASL
>>> >>>>>> (unknown error)
>>> >>>>>>> Exception in thread "main"
>>> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException:
>>> >>>>>> KeeperErrorCode = ConnectionLoss for /
>>> >>>>>>>   at
>>> >>>>>>
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
>>> >>>>>>>   at
>>> >>>>>>
>>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>>> >>>>>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
>>> >>>>>>>   at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
>>> >>>>>>>   at TestZkReadOnly.run(TestZkReadOnly.java:26)
>>> >>>>>>>   at TestZkReadOnly.main(TestZkReadOnly.java:16)
>>> >>>>>>>
>>> >>>>>>> I'm attaching all the logs. The server and the client
are on the
>>> same
>>> >>>>>> machine QA-E8WIN11.
>>> >>>>>>>
>>> >>>>>>> Any help would be greatly appreciated!
>>> >>>>>>>
>>> >>>>>>> Thanks,
>>> >>>>>>> Benjamin Jaton
>>> >>>>>>> <zkclient.log><zkserver.log>
>>> >>>>>>
>>> >>>>>>
>>> >>>>>
>>> >>>
>>> >>>
>>> >>
>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message