zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Johnson <mark.john...@servicemesh.com>
Subject Re: Problem creating Ephemeral znodes
Date Wed, 20 Sep 2017 18:09:51 GMT
Think I found the problem.  In the code that determines the EphemeralType
it is looking at the owner (which is the client ID or connection ID):

 EphemeralType.java:

    public static EphemeralType get(long ephemeralOwner) {
        if (ephemeralOwner == CONTAINER_EPHEMERAL_OWNER) {
            return CONTAINER;
        }
        if (ephemeralOwner < 0) {
            return TTL;
        }
        return (ephemeralOwner == 0) ? VOID : NORMAL;
    }

However my connection ID is:

 header.getClientId(): -720548323429908480

This causes the code to think this is a TTL Ephemeral node instead of a
NORMAL Ephemeral node.

This also explains why this is random - if my client ID is non-negative
then the node gets added correctly.

Can someone verify that this is a bug?  Can a ticket be created for this?

Thanks in advance,
Mark


On Wed, Sep 20, 2017 at 1:17 AM, Mark Johnson <mark.johnson@servicemesh.com>
wrote:

> Hi Abraham,
>
> I believe I'm using nio - from the log:
>
> 2017-09-19 15:25:51,761 [myid:010000001020] - INFO
> [main:NIOServerCnxnFactory@673] - Configuring NIO connection handler with
> 10s sessionless connection timeout, 1 selector thread(s), 8 worker threads,
> and 64 kB direct buffers.
>
> And here's my zoo.cfg:
>
> tickTime=2000
> initLimit=100
> syncLimit=5
> dataDir=/var/lib/zookeeper
> clientPort=2181
> autopurge.snapRetainCount=3
> # Purge task interval in hours
> autopurge.purgeInterval=1
> minSessionTimeout=4000
> maxSessionTimeout=60000
> standaloneEnabled=false
> reconfigEnabled=true
> skipACL=yes
> dynamicConfigFile=/opt/zookeeper/conf/zoo.cfg.dynamic
>
> *However, the problem is not removal of the ephemeral nodes, it's that
> they're not getting created in the first place,*  When I use the CLI to
> create an ephemeral node via:
>
> create -e /testnode
>
> It's not marked as an ephemeral node and hangs around forever.  From stat
> /testnode:
>
> ephemeralOwner = 0x0
>
> Sometimes Zookeeper will come up and create ephemeral nodes and they work
> correctly (deleted after the connection is terminated).  But usually (with
> the version I'm using- 3.5.3-beta2) they are not created.
>
> Are you familiar with the code?  I've downloaded the source and have been
> looking for the place where the znodes are actually created but have not
> found that section.  If you can point me to the right area of the code I
> can probably troubleshoot this issue.
>
> Thanks for any light you can shed on this problem!
>
> Mark
>
>
> On Tue, Sep 19, 2017 at 6:18 PM, Abraham Fine <afine@apache.org> wrote:
>
>> Hi Mark-
>>
>> I'm assuming you are using nio and not netty, please let me know if that
>> is not the case.
>>
>> Have you possibly set zookeeper.nio.sessionlessCnxnTimeout? This value
>> is 10s by default, so it may take a little more than ten seconds by
>> default for ephemeral nodes to be deleted.
>>
>> Thanks,
>> Abe
>>
>>
>> On Tue, Sep 19, 2017, at 08:16, Mark Johnson wrote:
>> > Hi,
>> >
>> > I'm running version 3.5.3-BETA of Zookeeper in a Docker container and
>> > usually cannot create Ephemeral znodes, either through the CLI or though
>> > a Java program.  It appears the owner is not set.  Here's an example
>> > session:
>> >
>> > ============================================================
>> ================================================
>> > [root@Appliance ~]# docker exec -it 9b7cd1e80b5b /bin/bash
>> > zookeeper@9b7cd1e80b5b:/opt/zookeeper-3.5.3-beta$ bin/zkCli.sh
>> > Connecting to localhost:2181
>> > 2017-09-19 10:03:31,373 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8
>> f21a47e06b42864475a60,
>> > built on 04/03/2017 16:19 GMT
>> > 2017-09-19 10:03:31,378 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:host.name=9b7cd1e80b5b
>> > 2017-09-19 10:03:31,378 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:java.version=1.8.0_144
>> > 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:java.vendor=Oracle Corporation
>> > 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:java.home=/usr/lib/jvm/java-8-oracle/jre
>> > 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:java.class.path=/opt/zookeeper-3.5.3-beta/bin/..
>> /build/classes:/opt/zookeeper-3.5.3-beta/bin/../build/lib/*.
>> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar
>> > :/opt/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.jar:/
>> opt/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Final.jar:/
>> opt/zookeeper-3.5.3-beta/bin/../lib/log4j-1.2.17.jar:/opt/
>> zookeeper-3.5.3-beta/bin/../lib/jline-2.11.jar:/opt/zookee
>> per-3.5.3-beta/bin/..
>> > /lib/jetty-util-9.2.18.v20160721.jar:/opt/zookeeper-3.5.3-
>> beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/opt/
>> zookeeper-3.5.3-beta/bin/../lib/jetty-server-9.2.18.v20160
>> 721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-security
>> -9.2.18.v20160721.jar:/op
>> > t/zookeeper-3.5.3-beta/bin/../lib/jetty-io-9.2.18.v20160721.
>> jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jetty-http-9.2.18.v
>> 20160721.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/javax.serv
>> let-api-3.1.0.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/
>> jackson-mapper-asl-1.
>> > 9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/jackson-core-a
>> sl-1.9.11.jar:/opt/zookeeper-3.5.3-beta/bin/../lib/commons-c
>> li-1.2.jar:/opt/zookeeper-3.5.3-beta/bin/../zookeeper-3.5.3-
>> beta.jar:/opt/zookeeper-3.5.3-beta/bin/../src/java/lib/*.jar
>> :/opt/zookeeper-
>> > 3.5.3-beta/bin/../conf:
>> > 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:java.library.path=/usr/java/packages/lib/amd64:/
>> usr/lib64:/lib64:/lib:/usr/lib
>> > 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:java.io.tmpdir=/tmp
>> > 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:java.compiler=<NA>
>> > 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:os.name=Linux
>> > 2017-09-19 10:03:31,380 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:os.arch=amd64
>> > 2017-09-19 10:03:31,381 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:os.version=3.10.0-514.26.2.el7.x86_64
>> > 2017-09-19 10:03:31,381 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:user.name=zookeeper
>> > 2017-09-19 10:03:31,381 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:user.home=/home/zookeeper
>> > 2017-09-19 10:03:31,381 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:user.dir=/opt/zookeeper-3.5.3-beta
>> > 2017-09-19 10:03:31,381 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:os.memory.free=233MB
>> > 2017-09-19 10:03:31,382 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:os.memory.max=240MB
>> > 2017-09-19 10:03:31,383 [myid:] - INFO  [main:Environment@109] - Client
>> > environment:os.memory.total=240MB
>> > 2017-09-19 10:03:31,386 [myid:] - INFO  [main:ZooKeeper@865] -
>> > Initiating client connection, connectString=localhost:2181
>> > sessionTimeout=30000
>> > watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@28ba21f3
>> > 2017-09-19 10:03:31,397 [myid:] - INFO  [main:ClientCnxnSocket@236] -
>> > jute.maxbuffer value is 4194304 Bytes
>> > 2017-09-19 10:03:31,410 [myid:localhost:2181] - INFO
>> > [main-SendThread(localhost:2181):ClientCnxn$SendThread@1113] - Opening
>> > socket connection to server localhost/127.0.0.1:2181. Will not attempt
>> > to authenticate using SASL (unknown error)
>> > Welcome to ZooKeeper!
>> > JLine support is enabled
>> > 2017-09-19 10:03:31,493 [myid:localhost:2181] - INFO
>> > [main-SendThread(localhost:2181):ClientCnxn$SendThread@948] - Socket
>> > connection established, initiating session, client: /127.0.0.1:44882,
>> > server: localhost/127.0.0.1:2181
>> > 2017-09-19 10:03:31,506 [myid:localhost:2181] - INFO
>> > [main-SendThread(localhost:2181):ClientCnxn$SendThread@1381] - Session
>> > establishment complete on server localhost/127.0.0.1:2181, sessionid =
>> > 0xeb0000480b3c005c, negotiated timeout = 30000
>> >
>> > WATCHER::
>> >
>> > WatchedEvent state:SyncConnected type:None path:null
>> > [zk: localhost:2181(CONNECTED) 0] create -e /testnode
>> > Created /testnode
>> > [zk: localhost:2181(CONNECTED) 1] stat /testnode
>> > cZxid = 0x2000002c4
>> > ctime = Tue Sep 19 10:03:50 CDT 2017
>> > mZxid = 0x2000002c4
>> > mtime = Tue Sep 19 10:03:50 CDT 2017
>> > pZxid = 0x2000002c4
>> > cversion = 0
>> > dataVersion = 0
>> > aclVersion = 0
>> > *ephemeralOwner = 0x0*
>> > dataLength = 0
>> > numChildren = 0
>> > [zk: localhost:2181(CONNECTED) 2]
>> > ============================================================
>> ==============================================
>> >
>> > If I disconnect and reconnect, the znode is still present as well. If I
>> > restart Zookeeper, many times this will resolve itself somehow and
>> > creation of ephemeral nodes will work correctly.  I have also tried this
>> > in a clustered configuration and get cases where one Zookeeper will
>> > create ephemeral nodes correctly and another Zookeeper will exhibit the
>> > above issue.
>> >
>> > Please let me know what other information would be of use or other
>> > things I could be looking for.
>> >
>> > Any help appreciated.
>> >
>> > Mark
>> >
>> >
>> >
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message