Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@zookeeper.apache.org
MIME-Version: 1.0
In-Reply-To: <60C9117D-F2E3-4235-9F56-CC8AFC51B98B@apache.org>
References: <CAPSuxQgCgQXdVzg5xMK9Y3L2h4osTF=kEQhtFW-2JW4fsBY9Lg@mail.gmail.com>
 <D392AAA0-354C-4F2C-91C6-CBF80706E65E@apache.org> <CAPSuxQg0auo1z4gRk_MKqqMwi_bG4e0+qAUpDkhHZksn=WzQmQ@mail.gmail.com>
 <CAPSuxQiwSNYXy2rUtrgiiCo0syX9E2cWzC0P4jRgh44k-BibwA@mail.gmail.com> <60C9117D-F2E3-4235-9F56-CC8AFC51B98B@apache.org>
From: Benjamin Jaton <benjamin.jaton@gmail.com>
Date: Thu, 28 Jul 2016 17:01:20 -0700
Message-ID: <CAPSuxQhioSusrL_27astB39oZs9D12RNtxL-KRv=n9QEoV2YTA@mail.gmail.com>
Subject: Re: ZK read-only issue
To: user@zookeeper.apache.org
Content-Type: multipart/alternative; boundary=001a114036c259816e0538baf3cd
archived-at: Fri, 29 Jul 2016 00:01:41 -0000

--001a114036c259816e0538baf3cd
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I will definitely try and I'll post an update, thanks again Flavio.

On Thu, Jul 28, 2016 at 4:59 PM, Flavio Junqueira <fpj@apache.org> wrote:

> I think you're hitting this:
>
> https://issues.apache.org/jira/browse/ZOOKEEPER-2466 <
> https://issues.apache.org/jira/browse/ZOOKEEPER-2466>
>
> which is resolved for 3.5.3 and trunk. You can try out the patch.
>
> -Flavio
>
> > On 29 Jul 2016, at 00:56, Benjamin Jaton <benjamin.jaton@gmail.com>
> wrote:
> >
> > OK so I did another fresh install this time using 3.5.2-alpha, and the
> read
> > only client doesn't seem to try all the hosts available.
> >
> > I create my client with:
> > ZooKeeper zk =3D new ZooKeeper("10.11.9.70:2181,10.11.12.4:2181,
> > 10.11.12.210:2181", 45000, this, true);
> >
> > In the following output, it only checks for 10.11.12.4:2181 and
> > 10.11.12.210:2181, but it doesn't try 10.11.9.70:2181 which is where th=
e
> > running server is.
> >
> > 0 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:zookeeper.version=3D3.5.2-alpha-1753710, built on 07/21/201=
6
> > 16:24 GMT
> > 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
> host.name
> > =3Dw-rli09-ben
> > 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:java.version=3D1.8.0_11
> > 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:java.vendor=3DOracle Corporation
> > 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:java.home=3D/usr/local/apps/jdk1.8.0_11/jre
> > 1 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> >
> environment:java.class.path=3D/home/benji/workspace-trunk/testzkclient/bi=
n:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/commons-cli-1.2.jar=
:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-core-asl-1.9=
.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jackson-mappe=
r-asl-1.9.11.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/java=
cc.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-6.1.26.j=
ar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jetty-util-6.1.26.=
jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/jline-2.11.jar:/h=
ome/benji/workspace-trunk/zookeeper-3.5.2/build/lib/log4j-1.2.17.jar:/home/=
benji/workspace-trunk/zookeeper-3.5.2/build/lib/netty-3.10.5.Final.jar:/hom=
e/benji/workspace-trunk/zookeeper-3.5.2/build/lib/servlet-api-2.5-20081211.=
jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-api-1.7.5.j=
ar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/lib/slf4j-log4j12-1.7.=
5.jar:/home/benji/workspace-trunk/zookeeper-3.5.2/build/zookeeper-3.5.2-alp=
ha.jar
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> >
> environment:java.library.path=3D/usr/java/packages/lib/amd64:/usr/lib64:/=
lib64:/lib:/usr/lib
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:java.io.tmpdir=3D/tmp
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:java.compiler=3D<NA>
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
> os.name
> > =3DLinux
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:os.arch=3Damd64
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:os.version=3D3.8.0-44-generic
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client environment:
> user.name
> > =3Dbenji
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:user.home=3D/home/benji
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:user.dir=3D/home/benji/workspace-trunk/testzkclient
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:os.memory.free=3D376MB
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:os.memory.max=3D5358MB
> > 2 [main] INFO org.apache.zookeeper.ZooKeeper  - Client
> > environment:os.memory.total=3D378MB
> > *8 [main] INFO org.apache.zookeeper.ZooKeeper  - Initiating client
> > connection, connectString=3D10.11.9.70:2181
> > <http://10.11.9.70:2181>,10.11.12.4:2181
> > <http://10.11.12.4:2181>,10.11.12.210:2181 <http://10.11.12.210:2181>
> > sessionTimeout=3D45000 watcher=3Dtestzkclient.TestZkReadOnlyClient@5010=
be6*
> > 21 [main] DEBUG org.apache.zookeeper.ClientCnxnSocket  - jute.maxbuffer
> is
> > 4194304
> > Successfully established the connection with ZooKeeper
> > *37 [main-SendThread(10.11.12.4:2181 <http://10.11.12.4:2181>)] INFO
> > org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> > 10.11.12.4/10.11.12.4:2181 <http://10.11.12.4/10.11.12.4:2181>. Will no=
t
> > attempt to authenticate using SASL (unknown error)*
> > 43 [main-SendThread(10.11.12.4:2181)] WARN
> org.apache.zookeeper.ClientCnxn
> > - Session 0x0 for server null, unexpected error, closing socket
> connection
> > and attempting reconnect
> > java.net.ConnectException: Connection refused
> >    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >    at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >    at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.=
java:357)
> >    at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> > 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> > org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
> > shutdown input
> > java.nio.channels.ClosedChannelException
> >    at
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >    at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >    at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java=
:198)
> >    at
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325=
)
> >    at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > 45 [main-SendThread(10.11.12.4:2181)] DEBUG
> > org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
> > shutdown output
> > java.nio.channels.ClosedChannelException
> >    at
> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >    at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >    at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java=
:205)
> >    at
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325=
)
> >    at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > *149 [main-SendThread(10.11.12.210:2181 <http://10.11.12.210:2181>)]
> INFO
> > org.apache.zookeeper.ClientCnxn  - Opening socket connection to server
> > 10.11.12.210/10.11.12.210:2181 <http://10.11.12.210/10.11.12.210:2181>.
> > Will not attempt to authenticate using SASL (unknown error)*
> > 150 [main-SendThread(10.11.12.210:2181)] WARN
> > org.apache.zookeeper.ClientCnxn  - Session 0x0 for server null,
> unexpected
> > error, closing socket connection and attempting reconnect
> > java.net.ConnectException: Connection refused
> >    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >    at
> > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
> >    at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.=
java:357)
> >    at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214)
> > 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> > org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
> > shutdown input
> > java.nio.channels.ClosedChannelException
> >    at
> > sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:775)
> >    at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:402)
> >    at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java=
:198)
> >    at
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325=
)
> >    at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > 150 [main-SendThread(10.11.12.210:2181)] DEBUG
> > org.apache.zookeeper.ClientCnxnSocketNIO  - Ignoring exception during
> > shutdown output
> > java.nio.channels.ClosedChannelException
> >    at
> > sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:792)
> >    at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:410)
> >    at
> >
> org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java=
:205)
> >    at
> > org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1325=
)
> >    at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1245)
> > *Exception in thread "main"
> > org.apache.zookeeper.KeeperException$ConnectionLossException:
> > KeeperErrorCode =3D ConnectionLoss for /*
> >    at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >    at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1956)
> >    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1985)
> >    at testzkclient.TestZkReadOnlyClient.run(TestZkReadOnlyClient.java:2=
6)
> >    at
> testzkclient.TestZkReadOnlyClient.main(TestZkReadOnlyClient.java:13)
> >
> >
> > Does anybody else have issues with the readonly mode?
> >
> > Thanks
> > Benjamin
> >
> >
> > On Sat, Jul 16, 2016 at 7:39 PM, Benjamin Jaton <
> benjamin.jaton@gmail.com>
> > wrote:
> >
> >> Very interesting, thank you Flavio.
> >>
> >> That should not happen because I have actually never started (even nev=
er
> >> created) the other servers. This server is the first one the client
> could
> >> have possibly ever seen. Maybe the server has to have been in a
> functioning
> >> ensemble at least once before being able to be read only able?
> >> I will recheck all my setup and do it again.
> >> Thanks again,
> >> Ben
> >>
> >> On Wed, Jul 13, 2016 at 2:06 PM, Flavio Junqueira <fpj@apache.org>
> wrote:
> >>
> >>> From the logs, it looks like the server the client is trying to conne=
ct
> >>> to is behind compared to last zxid the client has seen:
> >>>
> >>>        Refusing session request for client /10.11.12.4:49237 as it ha=
s
> >>> seen zxid 0x3 our last zxid is 0x0 client must try another server
> >>>
> >>> I believe the behavior we currently have is that a client can connect
> to
> >>> an RO server, but the server must have a view at least as recent as t=
he
> >>> server.
> >>>
> >>> -Flavio
> >>>
> >>>> On 12 Jul 2016, at 16:23, Benjamin Jaton <benjamin.jaton@gmail.com>
> >>> wrote:
> >>>>
> >>>> Hello,
> >>>>
> >>>> I'm having problems using ZK in read-only mode.
> >>>> I am using a fresh install of ZK 3.5.1-alpha where I configure the
> >>> installation to be in a 3 nodes ensemble.
> >>>> I start this node only (QA-E8WIN11) with -Dreadonlymode.enabled=3Dtr=
ue.
> >>>>
> >>>> I see that the node QA-E8WIN11 starts the RO server properly:
> >>>> 2016-07-12 16:10:10,955 [myid:1] - INFO
> >>> [Thread-4:ReadOnlyZooKeeperServer@73] - Read-only server started
> >>>>
> >>>> Then I make a read-only enabled client:
> >>>>
> >>>>        ZooKeeper zk =3D new
> >>> ZooKeeper("QA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181", 45000,
> this,
> >>> true);
> >>>>        System.out.println("Successfully established the connection
> >>> with ZooKeeper");
> >>>>
> >>>>        zk.getData("/", false, null);
> >>>>        System.out.println("Done.");
> >>>>
> >>>> Note that only QA-E8WIN11 is started, the other 2 are not.
> >>>>
> >>>> When the ZK client picks QA-E8WIN11 first, it works.
> >>>> But when it picks a different node, it fails with:
> >>>>
> >>>> 16:10:13.338 org.apache.zookeeper.ZooKeeper:716 - Initiating client
> >>> connection,
> connectString=3DQA-E8WIN12:2181,QA-E8WIN13:2181,QA-E8WIN11:2181
> >>> sessionTimeout=3D45000 watcher=3DTestZkReadOnly@68de145
> >>>> Successfully established the connection with ZooKeeper
> >>>> 16:10:13.475 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
> >>> connection to server QA-E8WIN13/10.11.8.230:2181 <
> >>> http://10.11.8.230:2181/>. Will not attempt to authenticate using SAS=
L
> >>> (unknown error)
> >>>> 16:10:28.469 org.apache.zookeeper.ClientCnxn:1251 - Client session
> >>> timed out, have not heard from server in 15001ms for sessionid 0x0,
> closing
> >>> socket connection and attempting reconnect
> >>>> 16:10:28.574 org.apache.zookeeper.ClientCnxn:1138 - Opening socket
> >>> connection to server QA-E8WIN12/10.11.8.232:2181 <
> >>> http://10.11.8.232:2181/>. Will not attempt to authenticate using SAS=
L
> >>> (unknown error)
> >>>> Exception in thread "main"
> >>> org.apache.zookeeper.KeeperException$ConnectionLossException:
> >>> KeeperErrorCode =3D ConnectionLoss for /
> >>>>    at
> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:99)
> >>>>    at
> >>> org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> >>>>    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1753)
> >>>>    at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1782)
> >>>>    at TestZkReadOnly.run(TestZkReadOnly.java:26)
> >>>>    at TestZkReadOnly.main(TestZkReadOnly.java:16)
> >>>>
> >>>> I'm attaching all the logs. The server and the client are on the sam=
e
> >>> machine QA-E8WIN11.
> >>>>
> >>>> Any help would be greatly appreciated!
> >>>>
> >>>> Thanks,
> >>>> Benjamin Jaton
> >>>> <zkclient.log><zkserver.log>
> >>>
> >>>
> >>
>
>

--001a114036c259816e0538baf3cd--