Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 495A4200B50 for ; Fri, 29 Jul 2016 21:07:58 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 47E66160A79; Fri, 29 Jul 2016 19:07:58 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 3F960160A6E for ; Fri, 29 Jul 2016 21:07:57 +0200 (CEST) Received: (qmail 76116 invoked by uid 500); 29 Jul 2016 19:07:56 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 76105 invoked by uid 99); 29 Jul 2016 19:07:56 -0000 Received: from mail-relay.apache.org (HELO mail-relay.apache.org) (140.211.11.15) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Jul 2016 19:07:56 +0000 Received: from mail-io0-f175.google.com (mail-io0-f175.google.com [209.85.223.175]) by mail-relay.apache.org (ASF Mail Server at mail-relay.apache.org) with ESMTPSA id D09521A003E for ; Fri, 29 Jul 2016 19:07:55 +0000 (UTC) Received: by mail-io0-f175.google.com with SMTP id q83so137596498iod.1 for ; Fri, 29 Jul 2016 12:07:55 -0700 (PDT) X-Gm-Message-State: AEkoouu4waYhKLdoCXcGauMIxpE1SE+VDKcjXpV4GgtlbZD8A44uv9X4xROWik0yn+UpvVhI6T6fVRm5t0r0Pw== X-Received: by 10.107.40.133 with SMTP id o127mr44045904ioo.183.1469819274999; Fri, 29 Jul 2016 12:07:54 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.216.37 with HTTP; Fri, 29 Jul 2016 12:07:54 -0700 (PDT) In-Reply-To: References: <60C9117D-F2E3-4235-9F56-CC8AFC51B98B@apache.org> From: Camille Fournier Date: Fri, 29 Jul 2016 15:07:54 -0400 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: ZK read-only issue To: "user@zookeeper.apache.org" Content-Type: multipart/alternative; boundary=001a11352e2ac5439b0538caf72b archived-at: Fri, 29 Jul 2016 19:07:58 -0000 --001a11352e2ac5439b0538caf72b Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Ok yeah I think this is reproducible and a bug in the client connection read-only logic. On Fri, Jul 29, 2016 at 2:43 PM, Camille Fournier wrote: > I'm looking at the readonly mode code right now and it appears that the > only way to set readonly mode is a global system property which means tha= t > the tests for this are only testing across 3 servers, all of which have > readonly mode set. So, this MAY be a bug, but what a pain to figure out h= ow > to reproduce. > > Global system properties: not even once. > > On Fri, Jul 29, 2016 at 12:27 PM, Flavio Junqueira wrote= : > >> Not that I'm aware of... >> >> -Flavio >> >> > On 29 Jul 2016, at 17:24, Benjamin Jaton >> wrote: >> > >> > Hi Flavio, >> > >> > The bug / patch is only for the C client, does this issue affect the >> java >> > client as well? >> > >> > On Thu, Jul 28, 2016 at 5:01 PM, Benjamin Jaton < >> benjamin.jaton@gmail.com> >> > wrote: >> > >> >> I will definitely try and I'll post an update, thanks again Flavio. >> >> >> >> On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira >> wrote: >> >> >> >>> I think you're hitting this: >> >>> >> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 < >> >>> https://issues.apache.org/jira/browse/ZOOKEEPER-2466> >> >>> >> >>> which is resolved for 3.5.3 and trunk. You can try out the patch. >> >>> >> >>> -Flavio >> >>> >> >>>> On 29 Jul 2016, at 00:56, Benjamin Jaton >> >>> wrote: >> >>>> >> >>>> OK so I did another fresh install this time using 3.5.2-alpha, and >> the >> >>> read >> >>>> only client doesn't seem to try all the hosts available. >> >>>> >> >>>> I create my client with: >> >>>> ZooKeeper zk =3D new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181, >> >>>> 10.11.12.210:2181", 45000, this, true); >> >>>> >> >>>> In the following output, it only checks for 10.11.12.4:2181 and >> >>>> 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is wher= e >> >>> the >> >>>> running server is. >> >>>> >> >>>> 0 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:zookeeper.version=3D3.5.2-alpha-1753710, built on >> 07/21/2016 >> >>>> 16:24 GMT >> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >> >>> host.name >> >>>> =3Dw-rli09-ben >> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:java.version=3D1.8.0_11 >> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:java.vendor=3DOracle Corporation >> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:java.home=3D/usr/local/apps/jdk1.8.0_11/jre >> >>>> 1 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> >> >>> >> environment:java.class.path=3D/home/benji/workspace-trunk/testzkclient/b= in:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.ja= r:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.= 9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mapp= er-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jav= acc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.= jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26= .jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/= home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home= /benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/ho= me/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211= .jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.= jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7= .5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-al= pha.jar >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> >> >>> >> environment:java.library.path=3D/usr/java/packages/lib/amd64:/usr/lib64:= /lib64:/lib:/usr/lib >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:java.io.tmpdir=3D/tmp >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:java.compiler=3D >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >> >>> os.name >> >>>> =3DLinux >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:os.arch=3Damd64 >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:os.version=3D3.8.0-44-generic >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment: >> >>> user.name >> >>>> =3Dbenji >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:user.home=3D/home/benji >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:user.dir=3D/home/benji/workspace-trunk/testzkclient >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:os.memory.free=3D376MB >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:os.memory.max=3D5358MB >> >>>> 2 [main] INFO org.apache.zookeeper.ZooKeeper - Client >> >>>> environment:os.memory.total=3D378MB >> >>>> *8 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client >> >>>> connection, connectString=3D10.11.9.70:2181 >> >>>> ,10.11.12.4:2181 >> >>>> ,10.11.12.210:2181 > > >> >>>> sessionTimeout=3D45000 >> watcher=3Dtestzkclient.TestZkReadOnlyClient@5010be6* >> >>>> 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket - >> jute.maxbuffer >> >>> is >> >>>> 4194304 >> >>>> Successfully established the connection with ZooKeeper >> >>>> *37 [main-SendThread(10.11.12.4:2181 )] INF= O >> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to >> server >> >>>> 10.11.12.4/10.11.12.4:2181 . Wil= l >> >>> not >> >>>> attempt to authenticate using SASL (unknown error)* >> >>>> 43 [main-SendThread(10.11.12.4:2181)] WARN >> >>> org.apache.zookeeper.ClientCnxn >> >>>> - Session 0x0 for server null, unexpected error, closing socket >> >>> connection >> >>>> and attempting reconnect >> >>>> java.net.ConnectException: Connection refused >> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> >>>> at >> >>>> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) >> >>>> at >> >>>> >> >>> >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO= .java:357) >> >>>> at >> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) >> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG >> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception duri= ng >> >>>> shutdown input >> >>>> java.nio.channels.ClosedChannelException >> >>>> at >> >>>> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) >> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) >> >>>> at >> >>>> >> >>> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.jav= a:198) >> >>>> at >> >>>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >> >>>> at >> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >> >>>> 45 [main-SendThread(10.11.12.4:2181)] DEBUG >> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception duri= ng >> >>>> shutdown output >> >>>> java.nio.channels.ClosedChannelException >> >>>> at >> >>>> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) >> >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410= ) >> >>>> at >> >>>> >> >>> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.jav= a:205) >> >>>> at >> >>>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >> >>>> at >> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >> >>>> *149 [main-SendThread(10.11.12.210:2181 )= ] >> >>> INFO >> >>>> org.apache.zookeeper.ClientCnxn - Opening socket connection to >> server >> >>>> 10.11.12.210/10.11.12.210:2181 < >> http://10.11.12.210/10.11.12.210:2181>. >> >>>> Will not attempt to authenticate using SASL (unknown error)* >> >>>> 150 [main-SendThread(10.11.12.210:2181)] WARN >> >>>> org.apache.zookeeper.ClientCnxn - Session 0x0 for server null, >> >>> unexpected >> >>>> error, closing socket connection and attempting reconnect >> >>>> java.net.ConnectException: Connection refused >> >>>> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) >> >>>> at >> >>>> >> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712) >> >>>> at >> >>>> >> >>> >> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO= .java:357) >> >>>> at >> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) >> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG >> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception duri= ng >> >>>> shutdown input >> >>>> java.nio.channels.ClosedChannelException >> >>>> at >> >>>> >> sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775) >> >>>> at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402) >> >>>> at >> >>>> >> >>> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.jav= a:198) >> >>>> at >> >>>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >> >>>> at >> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >> >>>> 150 [main-SendThread(10.11.12.210:2181)] DEBUG >> >>>> org.apache.zookeeper.ClientCnxnSocketNIO - Ignoring exception duri= ng >> >>>> shutdown output >> >>>> java.nio.channels.ClosedChannelException >> >>>> at >> >>>> >> sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792) >> >>>> at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410= ) >> >>>> at >> >>>> >> >>> >> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.jav= a:205) >> >>>> at >> >>>> >> org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325) >> >>>> at >> >>> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245) >> >>>> *Exception in thread "main" >> >>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >> >>>> KeeperErrorCode =3D ConnectionLoss for /* >> >>>> at >> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> >>>> at >> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956) >> >>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985) >> >>>> at >> >>> testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:26) >> >>>> at >> >>> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13) >> >>>> >> >>>> >> >>>> Does anybody else have issues with the readonly mode? >> >>>> >> >>>> Thanks >> >>>> Benjamin >> >>>> >> >>>> >> >>>> On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton < >> >>> benjamin.jaton@gmail.com> >> >>>> wrote: >> >>>> >> >>>>> Very interesting, thank you Flavio. >> >>>>> >> >>>>> That should not happen because I have actually never started (even >> >>> never >> >>>>> created) the other servers. This server is the first one the clien= t >> >>> could >> >>>>> have possibly ever seen. Maybe the server has to have been in a >> >>> functioning >> >>>>> ensemble at least once before being able to be read only able? >> >>>>> I will recheck all my setup and do it again. >> >>>>> Thanks again, >> >>>>> Ben >> >>>>> >> >>>>> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira >> >>> wrote: >> >>>>> >> >>>>>> From the logs, it looks like the server the client is trying to >> >>> connect >> >>>>>> to is behind compared to last zxid the client has seen: >> >>>>>> >> >>>>>> Refusing session request for client /10.11.12.4:49237 as it >> >>> has >> >>>>>> seen zxid 0x3 our last zxid is 0x0 client must try another server >> >>>>>> >> >>>>>> I believe the behavior we currently have is that a client can >> connect >> >>> to >> >>>>>> an RO server, but the server must have a view at least as recent = as >> >>> the >> >>>>>> server. >> >>>>>> >> >>>>>> -Flavio >> >>>>>> >> >>>>>>> On 12 Jul 2016, at 16:23, Benjamin Jaton < >> benjamin.jaton@gmail.com> >> >>>>>> wrote: >> >>>>>>> >> >>>>>>> Hello, >> >>>>>>> >> >>>>>>> I'm having problems using ZK in read-only mode. >> >>>>>>> I am using a fresh install of ZK 3.5.1-alpha where I configure t= he >> >>>>>> installation to be in a 3 nodes ensemble. >> >>>>>>> I start this node only (QA-E8WIN11) with >> -Dreadonlymode.enabled=3Dtrue. >> >>>>>>> >> >>>>>>> I see that the node QA-E8WIN11 starts the RO server properly: >> >>>>>>> 2016-07-12 16:10:10,955 [myid:1] - INFO >> >>>>>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started >> >>>>>>> >> >>>>>>> Then I make a read-only enabled client: >> >>>>>>> >> >>>>>>> ZooKeeper zk =3D new >> >>>>>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 4500= 0, >> >>> this, >> >>>>>> true); >> >>>>>>> System.out.println("Successfully established the connectio= n >> >>>>>> with ZooKeeper"); >> >>>>>>> >> >>>>>>> zk.getData("/", false, null); >> >>>>>>> System.out.println("Done."); >> >>>>>>> >> >>>>>>> Note that only QA-E8WIN11 is started, the other 2 are not. >> >>>>>>> >> >>>>>>> When the ZK client picks QA-E8WIN11 first, it works. >> >>>>>>> But when it picks a different node, it fails with: >> >>>>>>> >> >>>>>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating >> client >> >>>>>> connection, >> >>> connectString=3DQA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181 >> >>>>>> sessionTimeout=3D45000 watcher=3DTestZkReadOnly@68de145 >> >>>>>>> Successfully established the connection with ZooKeeper >> >>>>>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening sock= et >> >>>>>> connection to server QA-E8WIN13/10.11.8.230:2181 < >> >>>>>> http://10.11.8.230:2181/>. Will not attempt to authenticate using >> >>> SASL >> >>>>>> (unknown error) >> >>>>>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client sessi= on >> >>>>>> timed out, have not heard from server in 15001ms for sessionid 0x= 0, >> >>> closing >> >>>>>> socket connection and attempting reconnect >> >>>>>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening sock= et >> >>>>>> connection to server QA-E8WIN12/10.11.8.232:2181 < >> >>>>>> http://10.11.8.232:2181/>. Will not attempt to authenticate using >> >>> SASL >> >>>>>> (unknown error) >> >>>>>>> Exception in thread "main" >> >>>>>> org.apache.zookeeper.KeeperException$ConnectionLossException: >> >>>>>> KeeperErrorCode =3D ConnectionLoss for / >> >>>>>>> at >> >>>>>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:99) >> >>>>>>> at >> >>>>>> >> org.apache.zookeeper.KeeperException.create(KeeperException.java:51) >> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753) >> >>>>>>> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782) >> >>>>>>> at TestZkReadOnly.run(TestZkReadOnly.java:26) >> >>>>>>> at TestZkReadOnly.main(TestZkReadOnly.java:16) >> >>>>>>> >> >>>>>>> I'm attaching all the logs. The server and the client are on the >> same >> >>>>>> machine QA-E8WIN11. >> >>>>>>> >> >>>>>>> Any help would be greatly appreciated! >> >>>>>>> >> >>>>>>> Thanks, >> >>>>>>> Benjamin Jaton >> >>>>>>> >> >>>>>> >> >>>>>> >> >>>>> >> >>> >> >>> >> >> >> >> > --001a11352e2ac5439b0538caf72b--