zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jordan Zimmerman <jor...@jordanzimmerman.com>
Subject Re: Problem creating Ephemeral znodes
Date Wed, 20 Sep 2017 18:13:40 GMT
I didn't think client ID could be negative. Please open an issue in Jira and assign this to
me.

-Jordan

> On Sep 20, 2017, at 1:09 PM, Mark Johnson <mark.johnson@servicemesh.com> wrote:
> 
> Think I found the problem.  In the code that determines the EphemeralType
> it is looking at the owner (which is the client ID or connection ID):
> 
> EphemeralType.java:
> 
>    public static EphemeralType get(long ephemeralOwner) {
>        if (ephemeralOwner == CONTAINER_EPHEMERAL_OWNER) {
>            return CONTAINER;
>        }
>        if (ephemeralOwner < 0) {
>            return TTL;
>        }
>        return (ephemeralOwner == 0) ? VOID : NORMAL;
>    }
> 
> However my connection ID is:
> 
> header.getClientId(): -720548323429908480
> 
> This causes the code to think this is a TTL Ephemeral node instead of a
> NORMAL Ephemeral node.
> 
> This also explains why this is random - if my client ID is non-negative
> then the node gets added correctly.
> 
> Can someone verify that this is a bug?  Can a ticket be created for this?
> 
> Thanks in advance,
> Mark
> 
> 
> On Wed, Sep 20, 2017 at 1:17 AM, Mark Johnson <mark.johnson@servicemesh.com>
> wrote:
> 
>> Hi Abraham,
>> 
>> I believe I'm using nio - from the log:
>> 
>> 2017-09-19 15:25:51,761 [myid:010000001020] - INFO
>> [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with
>> 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads,
>> and 64 kB direct buffers.
>> 
>> And here's my zoo.cfg:
>> 
>> tickTime=2000
>> initLimit=100
>> syncLimit=5
>> dataDir=/var/lib/zookeeper
>> clientPort=2181
>> autopurge.snapRetainCount=3
>> # Purge task interval in hours
>> autopurge.purgeInterval=1
>> minSessionTimeout=4000
>> maxSessionTimeout=60000
>> standaloneEnabled=false
>> reconfigEnabled=true
>> skipACL=yes
>> dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic
>> 
>> *However, the problem is not removal of the ephemeral nodes, it's that
>> they're not getting created in the first place,*  When I use the CLI to
>> create an ephemeral node via:
>> 
>> create -e /testnode
>> 
>> It's not marked as an ephemeral node and hangs around forever.  From stat
>> /testnode:
>> 
>> ephemeralOwner = 0x0
>> 
>> Sometimes Zookeeper will come up and create ephemeral nodes and they work
>> correctly (deleted after the connection is terminated).  But usually (with
>> the version I'm using- 3.5.3-beta2) they are not created.
>> 
>> Are you familiar with the code?  I've downloaded the source and have been
>> looking for the place where the znodes are actually created but have not
>> found that section.  If you can point me to the right area of the code I
>> can probably troubleshoot this issue.
>> 
>> Thanks for any light you can shed on this problem!
>> 
>> Mark
>> 
>> 
>> On Tue, Sep 19, 2017 at 6:18 PM, Abraham Fine <afine@apache.org> wrote:
>> 
>>> Hi Mark-
>>> 
>>> I'm assuming you are using nio and not netty, please let me know if that
>>> is not the case.
>>> 
>>> Have you possibly set zookeeper.nio.sessionlessCnxnTimeout? This value
>>> is 10s by default, so it may take a little more than ten seconds by
>>> default for ephemeral nodes to be deleted.
>>> 
>>> Thanks,
>>> Abe
>>> 
>>> 
>>> On Tue, Sep 19, 2017, at 08:16, Mark Johnson wrote:
>>>> Hi,
>>>> 
>>>> I'm running version 3.5.3-BETA of Zookeeper in a Docker container and
>>>> usually cannot create Ephemeral znodes, either through the CLI or though
>>>> a Java program.  It appears the owner is not set.  Here's an example
>>>> session:
>>>> 
>>>> ============================================================
>>> ================================================
>>>> [root@Appliance ~]# docker exec -it 9b7cd1e80b5b /bin/bash
>>>> zookeeper@9b7cd1e80b5b:/opt/zookeeper-3.5.3-beta$ bin/zkCli.sh
>>>> Connecting to localhost:2181
>>>> 2017-09-19 10:03:31,373 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8
>>> f21a47e06b42864475a60,
>>>> built on 04/03/2017 16:19 GMT
>>>> 2017-09-19 10:03:31,378 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:host.name=9b7cd1e80b5b
>>>> 2017-09-19 10:03:31,378 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:java.version=1.8.0_144
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:java.vendor=Oracle Corporation
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:java.class.path=/opt/zookeeper-3.5.3-beta/bin/..
>>> /build/classes:/opt/zookeeper-3.5.3-beta/bin/../build/lib/*.
>>> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar
>>>> :/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.jar:/
>>> opt/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Final.jar:/
>>> opt/zookeeper-3.5.3-beta/bin/../lib/log4j-1.2.17.jar:/opt/
>>> zookeeper-3.5.3-beta/bin/../lib/jline-2.11.jar:/opt/zookee
>>> per-3.5.3-beta/bin/..
>>>> /lib/jetty-util-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-
>>> beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/opt/
>>> zookeeper-3.5.3-beta/bin/../lib/jetty-server-9.2.18.v20160
>>> 721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-security
>>> -9.2.18.v20160721.jar:/op
>>>> t/zookeeper-3.5.3-beta/bin/../lib/jetty-io-9.2.18.v20160721.
>>> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-http-9.2.18.v
>>> 20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/javax.serv
>>> let-api-3.1.0.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/
>>> jackson-mapper-asl-1.
>>>> 9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jackson-core-a
>>> sl-1.9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/commons-c
>>> li-1.2.jar:/opt/zookeeper-3.5.3-beta/bin/../zookeeper-3.5.3-
>>> beta.jar:/opt/zookeeper-3.5.3-beta/bin/../src/java/lib/*.jar
>>> :/opt/zookeeper-
>>>> 3.5.3-beta/bin/../conf:
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:java.library.path=/usr/java/packages/lib/amd64:/
>>> usr/lib64:/lib64:/lib:/usr/lib
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:java.io.tmpdir=/tmp
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:java.compiler=<NA>
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:os.name=Linux
>>>> 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:os.arch=amd64
>>>> 2017-09-19 10:03:31,381 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:os.version=3.10.0-514.26.2.el7.x86_64
>>>> 2017-09-19 10:03:31,381 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:user.name=zookeeper
>>>> 2017-09-19 10:03:31,381 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:user.home=/home/zookeeper
>>>> 2017-09-19 10:03:31,381 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:user.dir=/opt/zookeeper-3.5.3-beta
>>>> 2017-09-19 10:03:31,381 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:os.memory.free=233MB
>>>> 2017-09-19 10:03:31,382 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:os.memory.max=240MB
>>>> 2017-09-19 10:03:31,383 [myid:] - INFO  [main:Environment@109] - Client
>>>> environment:os.memory.total=240MB
>>>> 2017-09-19 10:03:31,386 [myid:] - INFO  [main:ZooKeeper@865] -
>>>> Initiating client connection, connectString=localhost:2181
>>>> sessionTimeout=30000
>>>> watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@28ba21f3
>>>> 2017-09-19 10:03:31,397 [myid:] - INFO  [main:ClientCnxnSocket@236] -
>>>> jute.maxbuffer value is 4194304 Bytes
>>>> 2017-09-19 10:03:31,410 [myid:localhost:2181] - INFO
>>>> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1113] - Opening
>>>> socket connection to server localhost/127.0.0.1:2181. Will not attempt
>>>> to authenticate using SASL (unknown error)
>>>> Welcome to ZooKeeper!
>>>> JLine support is enabled
>>>> 2017-09-19 10:03:31,493 [myid:localhost:2181] - INFO
>>>> [main-SendThread(localhost:2181):ClientCnxn$SendThread@948] - Socket
>>>> connection established, initiating session, client: /127.0.0.1:44882,
>>>> server: localhost/127.0.0.1:2181
>>>> 2017-09-19 10:03:31,506 [myid:localhost:2181] - INFO
>>>> [main-SendThread(localhost:2181):ClientCnxn$SendThread@1381] - Session
>>>> establishment complete on server localhost/127.0.0.1:2181, sessionid =
>>>> 0xeb0000480b3c005c, negotiated timeout = 30000
>>>> 
>>>> WATCHER::
>>>> 
>>>> WatchedEvent state:SyncConnected type:None path:null
>>>> [zk: localhost:2181(CONNECTED) 0] create -e /testnode
>>>> Created /testnode
>>>> [zk: localhost:2181(CONNECTED) 1] stat /testnode
>>>> cZxid = 0x2000002c4
>>>> ctime = Tue Sep 19 10:03:50 CDT 2017
>>>> mZxid = 0x2000002c4
>>>> mtime = Tue Sep 19 10:03:50 CDT 2017
>>>> pZxid = 0x2000002c4
>>>> cversion = 0
>>>> dataVersion = 0
>>>> aclVersion = 0
>>>> *ephemeralOwner = 0x0*
>>>> dataLength = 0
>>>> numChildren = 0
>>>> [zk: localhost:2181(CONNECTED) 2]
>>>> ============================================================
>>> ==============================================
>>>> 
>>>> If I disconnect and reconnect, the znode is still present as well. If I
>>>> restart Zookeeper, many times this will resolve itself somehow and
>>>> creation of ephemeral nodes will work correctly.  I have also tried this
>>>> in a clustered configuration and get cases where one Zookeeper will
>>>> create ephemeral nodes correctly and another Zookeeper will exhibit the
>>>> above issue.
>>>> 
>>>> Please let me know what other information would be of use or other
>>>> things I could be looking for.
>>>> 
>>>> Any help appreciated.
>>>> 
>>>> Mark
>>>> 
>>>> 
>>>> 
>>> 
>> 
>> 


Mime
View raw message