hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: RpcRetryingCaller error accessing HBase from MapReduce job
Date Wed, 25 Jan 2017 14:23:34 GMT
bq.     <name>hbase.zookeeper.property.server.7</name>

I searched 1.2 codebase but didn't find config parameter in the above form.
http://hbase.apache.org/book.html didn't mention it either.

May I ask where you obtained such config ?

For hbase.zookeeper.quorum, do you have zookeeper running on the 12 nodes ?
Normally 3 zookeeper nodes should be enough.

Cheers

On Wed, Jan 25, 2017 at 2:08 AM, Hernán Blanco <hernanblancolanda@gmail.com>
wrote:

> Hi all,
>
> I'm running HBase 1.2.4 on a distributed setup with 12 virtual machines
> on the same local network. The "main" node (node26.example.com) runs the
> HMaster, while the other 11 machines run RegionServers. No backup
> HMaster. This cluster also runs Hadoop 2.7.2 smoothly.
>
> Both HBase shell and accessing through the HBase client API run
> properly, and even importing a TSV file into a table with
> org.apache.hadoop.hbase.mapreduce.ImportTsv succeeds, remaining
> registered in the YARN history.
>
> But the problem appears when trying to access from a MapReduce job to an
> HBase table (using Hadoop 2.7.2). Here a minimal code that produces the
> issue by connecting and scanning an HBase table:
> http://pastebin.com/pm8tbbTq. The maps hang until timeout and then
> retries deploying new maps until failing, each map showing the following
> messages in the syslog:
>
> ----------
>
> 2017-01-24 20:04:07,904 INFO [main] org.apache.hadoop.metrics2.impl.MetricsConfig:
> loaded properties from hadoop-metrics2.properties
> 2017-01-24 20:04:08,186 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
> Scheduled snapshot period at 10 second(s).
> 2017-01-24 20:04:08,186 INFO [main] org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
> MapTask metrics system started
> 2017-01-24 20:04:08,193 INFO [main] org.apache.hadoop.mapred.YarnChild:
> Executing with tokens:
> 2017-01-24 20:04:08,193 INFO [main] org.apache.hadoop.mapred.YarnChild:
> Kind: mapreduce.job, Service: job_1485182272940_0336, Ident:
> (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@6f03482)
> 2017-01-24 20:04:08,450 INFO [main] org.apache.hadoop.mapred.YarnChild:
> Sleeping for 0ms before retrying again. Got null now.
> 2017-01-24 20:04:08,994 INFO [main] org.apache.hadoop.mapred.YarnChild:
> mapreduce.cluster.local.dir for child: /tmp/hadoop-hdadmin/nm-local-
> dir/usercache/idstest/appcache/application_1485182272940_0336
> 2017-01-24 20:04:09,617 INFO [main] org.apache.hadoop.conf.Configuration.deprecation:
> session.id is deprecated. Instead, use dfs.metrics.session-id
> 2017-01-24 20:04:10,968 INFO [main] org.apache.hadoop.mapreduce.
> lib.output.FileOutputCommitter: File Output Committer Algorithm version
> is 1
> 2017-01-24 20:04:11,044 INFO [main] org.apache.hadoop.mapred.Task:  Using
> ResourceCalculatorProcessTree : [ ]
> 2017-01-24 20:04:11,455 INFO [main] org.apache.hadoop.mapred.MapTask:
> Processing split: hdfs://node26.example.com/user/idstest/data/articles-50/
> 99322.txt:0+3757
> 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask:
> (EQUATOR) 0 kvi 26214396(104857584)
> 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask:
> mapreduce.task.io.sort.mb: 100
> 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask:
> soft limit at 83886080
> 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask:
> bufstart = 0; bufvoid = 104857600
> 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask:
> kvstart = 26214396; length = 6553600
> 2017-01-24 20:04:11,612 INFO [main] org.apache.hadoop.mapred.MapTask: Map
> output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
> 2017-01-24 20:04:12,211 INFO [main] org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper:
> Process identifier=hconnection-0x650eab8 connecting to ZooKeeper
> ensemble=localhost:2181
> 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014
> 09:09 GMT
> 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:host.name=node27.example.com
> 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:java.version=1.8.0_77-Debian
> 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:java.vendor=Oracle Corporation
> 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
> 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:java.class.path= /* ... */
> 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:java.library.path=/tmp/hadoop-hdadmin/nm-local-
> dir/usercache/idstest/appcache/application_1485182272940_0336/container_
> 1485182272940_0336_01_000048:/opt/hadoop-2.7.2/lib/native:/
> usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/
> lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
> 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:java.io.tmpdir=/tmp/hadoop-hdadmin/nm-local-
> dir/usercache/idstest/appcache/application_1485182272940_0336/container_
> 1485182272940_0336_01_000048/tmp
> 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:java.compiler=<NA>
> 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:os.name=Linux
> 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:os.arch=amd64
> 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:os.version=3.16.0-4-amd64
> 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:user.name=hdadmin
> 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:user.home=/home/hdadmin
> 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> Client environment:user.dir=/tmp/hadoop-hdadmin/nm-local-dir/
> usercache/idstest/appcache/application_1485182272940_
> 0336/container_1485182272940_0336_01_000048
> 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> Initiating client connection, connectString=localhost:2181
> sessionTimeout=90000 watcher=hconnection-0x650eab80x0,
> quorum=localhost:2181, baseZNode=/hbase
> 2017-01-24 20:04:12,322 INFO [main-SendThread(node27.example.com:2181)]
> org.apache.zookeeper.ClientCnxn: Opening socket connection to server
> node27.example.com/192.168.0.17:2181. Will not attempt to authenticate
> using SASL (unknown error)
> 2017-01-24 20:04:12,323 INFO [main-SendThread(node27.example.com:2181)]
> org.apache.zookeeper.ClientCnxn: Socket connection established to
> node27.example.com/192.168.0.17:2181, initiating session
> 2017-01-24 20:04:12,331 INFO [main-SendThread(node27.example.com:2181)]
> org.apache.zookeeper.ClientCnxn: Session establishment complete on server
> node27.example.com/192.168.0.17:2181, sessionid = 0x159d07980b50189,
> negotiated timeout = 90000
> 2017-01-24 20:04:51,657 INFO [hconnection-0x650eab8-metaLookup-shared--pool2-t1]
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=10, retries=35, started=38852 ms ago, cancelled=false, msg=row
> 'idstest1,,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=node149.example.com,16020,1485261342943, seqNum=0
> 2017-01-24 20:05:01,664 INFO [hconnection-0x650eab8-metaLookup-shared--pool2-t1]
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=11, retries=35, started=48860 ms ago, cancelled=false, msg=row
> 'idstest1,,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=node149.example.com,16020,1485261342943, seqNum=0
> 2017-01-24 20:05:40,044 INFO [hconnection-0x650eab8-metaLookup-shared--pool2-t2]
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=10, retries=35, started=38273 ms ago, cancelled=false, msg=row
> 'idstest1,,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=node149.example.com,16020,1485261342943, seqNum=0
> 2017-01-24 20:05:50,059 INFO [hconnection-0x650eab8-metaLookup-shared--pool2-t2]
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=11, retries=35, started=48288 ms ago, cancelled=false, msg=row
> 'idstest1,,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=node149.example.com,16020,1485261342943, seqNum=0
> 2017-01-24 20:06:28,543 INFO [hconnection-0x650eab8-metaLookup-shared--pool2-t3]
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=10, retries=35, started=38273 ms ago, cancelled=false, msg=row
> 'idstest1,,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=node149.example.com,16020,1485261342943, seqNum=0
> 2017-01-24 20:06:38,632 INFO [hconnection-0x650eab8-metaLookup-shared--pool2-t3]
> org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> tries=11, retries=35, started=48362 ms ago, cancelled=false, msg=row
> 'idstest1,,99999999999999' on table 'hbase:meta' at region=hbase:meta,,1.1588230740,
> hostname=node149.example.com,16020,1485261342943, seqNum=0
> ...
>
> ----------
>
> My hbase-site.xml in the Master node node26.example.com (almost same for
> the other nodes, just referring as 0.0.0.0 to themselves):
>
> ----------
>
> <configuration>
>   <property>
>     <name>hbase.rootdir</name>
>     <value>hdfs://node26.example.com:8020/hbase</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.dataDir</name>
>     <value>/home/hdadmin/zookeeper</value>
>   </property>
>   <property>
>     <name>hbase.cluster.distributed</name>
>     <value>true</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.quorum</name>
>     <value>node26.example.com,node27.example.com,node144.example.com,
> node145.example.com,node146.example.com,node147.example.com,
> node148.example.com,node149.example.com,node150.example.com,
> node151.example.com,node152.example.com,node153.example.com</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.0</name>
>     <value>0.0.0.0:2888:3888</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.1</name>
>     <value>node27.example.com:2888:3888</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.2</name>
>     <value>node144.example.com:2888:3888</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.3</name>
>     <value>node145.example.com:2888:3888</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.4</name>
>     <value>node146.example.com:2888:3888</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.5</name>
>     <value>node147.example.com:2888:3888</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.6</name>
>     <value>node148.example.com:2888:3888</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.7</name>
>     <value>node149.example.com:2888:3888</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.8</name>
>     <value>node150.example.com:2888:3888</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.9</name>
>     <value>node151.example.com:2888:3888</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.10</name>
>     <value>node152.example.com:2888:3888</value>
>   </property>
>   <property>
>     <name>hbase.zookeeper.property.server.11</name>
>     <value>node153.example.com:2888:3888</value>
>   </property>
> </configuration>
>
> ----------
>
> After starting HBase using the start-hbase.sh script, the RegionServer
> logs have no warnings or errors, but the different ZooKeeper instances
> generate diverse outputs. 3 examples:
>
>
> - The master node has an empty log:
>
> ----------
>
> Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node26.example.com
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 128914
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 65536
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 128914
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
>
> ----------
>
>
> - node153.example.com has the following small ZooKeeper log:
>
> ----------
>
> Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node153.example.com
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 128914
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 65536
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 128914
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> 2017-01-24 20:52:39,816 WARN  [main] quorum.QuorumPeerConfig: Non-optimial
> configuration, consider an odd number of servers.
> 2017-01-24 20:52:39,817 INFO  [main] quorum.QuorumPeerConfig: Defaulting
> to majority quorums
> 2017-01-24 20:52:39,984 INFO  [main] quorum.QuorumPeerMain: Starting
> quorum peer
> 2017-01-24 20:52:39,994 INFO  [main] server.NIOServerCnxnFactory: binding
> to port 0.0.0.0/0.0.0.0:2181
>
> ----------
>
> As an extra hint, this node is not generating
> the /tmp/hbase-hdadmin-zookeeper.pid, so it shows an error when running
> the stop-hbase.sh script.
>
>
> - And node144.example.com (as other nodes) has the following ZooKeeper
> log with "connection refused" errors:
>
> ----------
>
> Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node144.example.com
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 128914
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 65536
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 128914
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> 2017-01-24 20:52:39,473 WARN  [main] quorum.QuorumPeerConfig: Non-optimial
> configuration, consider an odd number of servers.
> 2017-01-24 20:52:39,473 INFO  [main] quorum.QuorumPeerConfig: Defaulting
> to majority quorums
> 2017-01-24 20:52:39,567 INFO  [main] quorum.QuorumPeerMain: Starting
> quorum peer
> 2017-01-24 20:52:39,581 INFO  [main] server.NIOServerCnxnFactory: binding
> to port 0.0.0.0/0.0.0.0:2181
> 2017-01-24 20:52:39,601 INFO  [main] quorum.QuorumPeer: tickTime set to
> 3000
> 2017-01-24 20:52:39,601 INFO  [main] quorum.QuorumPeer: minSessionTimeout
> set to -1
> 2017-01-24 20:52:39,601 INFO  [main] quorum.QuorumPeer: maxSessionTimeout
> set to 90000
> 2017-01-24 20:52:39,601 INFO  [main] quorum.QuorumPeer: initLimit set to 10
> 2017-01-24 20:52:39,613 INFO  [main] persistence.FileSnap: Reading
> snapshot /home/hdadmin/zookeeper2/version-2/snapshot.1d0000023f
> 2017-01-24 20:52:39,714 INFO  [Thread-2] quorum.QuorumCnxManager: My
> election bind port: node144.example.com/192.168.0.19:3888
> 2017-01-24 20:52:39,729 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> quorum.QuorumPeer: LOOKING
> 2017-01-24 20:52:39,731 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> quorum.FastLeaderElection: New election. My id =  2, proposed
> zxid=0x1e0000258d
> 2017-01-24 20:52:39,740 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 2
> (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 2
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,740 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 0
> (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 0
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,742 WARN  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Cannot open channel to 3 at election address
> node145.example.com/192.168.0.18:3888
> java.net.ConnectException: Connection refused
>         at java.net.PlainSocketImpl.socketConnect(Native Method)
>         at java.net.AbstractPlainSocketImpl.doConnect(
> AbstractPlainSocketImpl.java:350)
>         at java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:206)
>         at java.net.AbstractPlainSocketImpl.connect(
> AbstractPlainSocketImpl.java:188)
>         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>         at java.net.Socket.connect(Socket.java:589)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> connectOne(QuorumCnxManager.java:368)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> toSend(QuorumCnxManager.java:341)
>         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> Messenger$WorkerSender.process(FastLeaderElection.java:449)
>         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> Messenger$WorkerSender.run(FastLeaderElection.java:430)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 20:52:39,744 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 1
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,745 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 1
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,745 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 2
> (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 0
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,746 INFO  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Have smaller server identifier, so dropping the
> connection: (4, 2)
> 2017-01-24 20:52:39,747 INFO  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Have smaller server identifier, so dropping the
> connection: (5, 2)
> 2017-01-24 20:52:39,747 INFO  [node144.example.com/192.168.0.19:3888]
> quorum.QuorumCnxManager: Received connection request /192.168.0.21:40064
> 2017-01-24 20:52:39,747 INFO  [node144.example.com/192.168.0.19:3888]
> quorum.QuorumCnxManager: Received connection request /192.168.0.22:54362
> 2017-01-24 20:52:39,747 INFO  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Have smaller server identifier, so dropping the
> connection: (6, 2)
> 2017-01-24 20:52:39,748 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 4
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,748 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 5
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,748 INFO  [node144.example.com/192.168.0.19:3888]
> quorum.QuorumCnxManager: Received connection request /192.168.0.23:53410
> 2017-01-24 20:52:39,749 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 5
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,749 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 0
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,749 INFO  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Have smaller server identifier, so dropping the
> connection: (7, 2)
> 2017-01-24 20:52:39,749 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 4
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,749 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 6
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,750 INFO  [node144.example.com/192.168.0.19:3888]
> quorum.QuorumCnxManager: Received connection request /192.168.0.25:33085
> 2017-01-24 20:52:39,750 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 6
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,751 WARN  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Cannot open channel to 8 at election address
> node150.example.com/192.168.0.24:3888
> java.net.ConnectException: Connection refused
>         at java.net.PlainSocketImpl.socketConnect(Native Method)
>         at java.net.AbstractPlainSocketImpl.doConnect(
> AbstractPlainSocketImpl.java:350)
>         at java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:206)
>         at java.net.AbstractPlainSocketImpl.connect(
> AbstractPlainSocketImpl.java:188)
>         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>         at java.net.Socket.connect(Socket.java:589)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> connectOne(QuorumCnxManager.java:368)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> toSend(QuorumCnxManager.java:341)
>         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> Messenger$WorkerSender.process(FastLeaderElection.java:449)
>         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> Messenger$WorkerSender.run(FastLeaderElection.java:430)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 20:52:39,751 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 7
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,752 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 7
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,753 INFO  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Have smaller server identifier, so dropping the
> connection: (9, 2)
> 2017-01-24 20:52:39,753 INFO  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Have smaller server identifier, so dropping the
> connection: (10, 2)
> 2017-01-24 20:52:39,753 INFO  [node144.example.com/192.168.0.19:3888]
> quorum.QuorumCnxManager: Received connection request /192.168.0.26:35997
> 2017-01-24 20:52:39,754 INFO  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Have smaller server identifier, so dropping the
> connection: (11, 2)
> 2017-01-24 20:52:39,755 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 2
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,755 INFO  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Have smaller server identifier, so dropping the
> connection: (3, 2)
> 2017-01-24 20:52:39,756 WARN  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Cannot open channel to 8 at election address
> node150.example.com/192.168.0.24:3888
> java.net.ConnectException: Connection refused
>         at java.net.PlainSocketImpl.socketConnect(Native Method)
>         at java.net.AbstractPlainSocketImpl.doConnect(
> AbstractPlainSocketImpl.java:350)
>         at java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:206)
>         at java.net.AbstractPlainSocketImpl.connect(
> AbstractPlainSocketImpl.java:188)
>         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>         at java.net.Socket.connect(Socket.java:589)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> connectOne(QuorumCnxManager.java:368)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> toSend(QuorumCnxManager.java:341)
>         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> Messenger$WorkerSender.process(FastLeaderElection.java:449)
>         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> Messenger$WorkerSender.run(FastLeaderElection.java:430)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 20:52:39,757 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 9
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,757 INFO  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Have smaller server identifier, so dropping the
> connection: (10, 2)
> 2017-01-24 20:52:39,757 INFO  [WorkerSender[myid=2]]
> quorum.QuorumCnxManager: Have smaller server identifier, so dropping the
> connection: (11, 2)
> 2017-01-24 20:52:39,758 INFO  [node144.example.com/192.168.0.19:3888]
> quorum.QuorumCnxManager: Received connection request /192.168.0.27:34404
> 2017-01-24 20:52:39,759 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 9
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,768 INFO  [node144.example.com/192.168.0.19:3888]
> quorum.QuorumCnxManager: Received connection request /192.168.0.18:59032
> 2017-01-24 20:52:39,769 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 11
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,769 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 3
> (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 3
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,776 INFO  [node144.example.com/192.168.0.19:3888]
> quorum.QuorumCnxManager: Received connection request /192.168.0.27:34405
> 2017-01-24 20:52:39,776 WARN  [RecvWorker:11] quorum.QuorumCnxManager:
> Connection broken for id 11, my id = 2, error =
> java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:765)
> 2017-01-24 20:52:39,777 WARN  [RecvWorker:11] quorum.QuorumCnxManager:
> Interrupting SendWorker
> 2017-01-24 20:52:39,777 WARN  [SendWorker:11] quorum.QuorumCnxManager:
> Interrupted while waiting for message on queue
> java.lang.InterruptedException
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> java:2014)
>         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
>         at java.util.concurrent.ArrayBlockingQueue.poll(
> ArrayBlockingQueue.java:418)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> pollSendQueue(QuorumCnxManager.java:849)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> access$500(QuorumCnxManager.java:64)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> SendWorker.run(QuorumCnxManager.java:685)
> 2017-01-24 20:52:39,777 INFO  [node144.example.com/192.168.0.19:3888]
> quorum.QuorumCnxManager: Received connection request /192.168.0.28:57974
> 2017-01-24 20:52:39,778 WARN  [SendWorker:11] quorum.QuorumCnxManager:
> Send worker leaving thread
> 2017-01-24 20:52:39,778 INFO  [node144.example.com/192.168.0.19:3888]
> quorum.QuorumCnxManager: Received connection request /192.168.0.28:57975
> 2017-01-24 20:52:39,778 WARN  [RecvWorker:10] quorum.QuorumCnxManager:
> Connection broken for id 10, my id = 2, error =
> java.io.EOFException
>         at java.io.DataInputStream.readInt(DataInputStream.java:392)
>         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> RecvWorker.run(QuorumCnxManager.java:765)
> 2017-01-24 20:52:39,778 WARN  [RecvWorker:10] quorum.QuorumCnxManager:
> Interrupting SendWorker
> 2017-01-24 20:52:39,778 WARN  [SendWorker:10] quorum.QuorumCnxManager:
> Exception when using channel: for id 10 my id = 2 error =
> java.net.SocketException: Broken pipe
> 2017-01-24 20:52:39,778 WARN  [SendWorker:10] quorum.QuorumCnxManager:
> Send worker leaving thread
> 2017-01-24 20:52:39,779 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 3
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,782 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 10
> (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 10
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,788 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 11
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,789 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 10
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,815 INFO  [node144.example.com/192.168.0.19:3888]
> quorum.QuorumCnxManager: Received connection request /192.168.0.24:56503
> 2017-01-24 20:52:39,829 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 8
> (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 8
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:39,829 INFO  [WorkerReceiver[myid=2]]
> quorum.FastLeaderElection: Notification: 1 (message format version), 11
> (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 8
> (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> 2017-01-24 20:52:40,030 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> quorum.QuorumPeer: FOLLOWING
> 2017-01-24 20:52:40,035 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> quorum.Learner: TCP NoDelay set to: true
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:zookeeper.version=3.4.6-1569965,
> built on 02/20/2014 09:09 GMT
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:host.name=node144.example.com
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:java.version=1.8.0_77-Debian
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:java.vendor=Oracle Corporation
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:java.home=/usr/
> lib/jvm/java-8-openjdk-amd64/jre
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:java.class.path= /* ... */
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:java.library.path=
> /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/
> lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:java.compiler=<NA>
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:os.name=Linux
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:os.arch=amd64
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:os.version=3.16.0-4-amd64
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:user.name=hdadmin
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:user.home=/home/hdadmin
> 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Server environment:user.dir=/opt/hbase-1.2.4
> 2017-01-24 20:52:40,042 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> server.ZooKeeperServer: Created server with tickTime 3000 minSessionTimeout
> 6000 maxSessionTimeout 90000 datadir /home/hdadmin/zookeeper2/version-2
> snapdir /home/hdadmin/zookeeper2/version-2
> 2017-01-24 20:52:40,043 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> quorum.Learner: FOLLOWING - LEADER ELECTION TOOK - 311
> 2017-01-24 20:52:40,046 WARN  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> quorum.Learner: Unexpected exception, tries=0, connecting to
> node153.example.com/192.168.0.27:2888
> java.net.ConnectException: Connection refused
>         at java.net.PlainSocketImpl.socketConnect(Native Method)
>         at java.net.AbstractPlainSocketImpl.doConnect(
> AbstractPlainSocketImpl.java:350)
>         at java.net.AbstractPlainSocketImpl.connectToAddress(
> AbstractPlainSocketImpl.java:206)
>         at java.net.AbstractPlainSocketImpl.connect(
> AbstractPlainSocketImpl.java:188)
>         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
>         at java.net.Socket.connect(Socket.java:589)
>         at org.apache.zookeeper.server.quorum.Learner.
> connectToLeader(Learner.java:225)
>         at org.apache.zookeeper.server.quorum.Follower.followLeader(
> Follower.java:71)
>         at org.apache.zookeeper.server.quorum.QuorumPeer.run(
> QuorumPeer.java:786)
> 2017-01-24 20:52:41,073 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> quorum.Learner: Getting a diff from the leader 0x1e0000258d
> 2017-01-24 20:52:41,116 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> persistence.FileTxnSnapLog: Snapshotting: 0x1e0000258d to
> /home/hdadmin/zookeeper2/version-2/snapshot.1e0000258d
> 2017-01-24 20:52:42,408 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.29:44336
> 2017-01-24 20:52:42,418 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.29:44336
> 2017-01-24 20:52:42,421 WARN  [QuorumPeer[myid=2]/0:0:0:0:0:0:0:0:2181]
> quorum.Learner: Got zxid 0x1f00000001 expected 0x1
> 2017-01-24 20:52:42,421 INFO  [SyncThread:2] persistence.FileTxnLog:
> Creating new log file: log.1f00000001
> 2017-01-24 20:52:42,440 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750000 with negotiated timeout 90000 for
> client /192.168.0.29:44336
> 2017-01-24 20:52:42,734 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.22:43377
> 2017-01-24 20:52:42,737 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.22:43377
> 2017-01-24 20:52:42,742 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750001 with negotiated timeout 90000 for
> client /192.168.0.22:43377
> 2017-01-24 20:52:43,257 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.23:59005
> 2017-01-24 20:52:43,258 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.23:59005
> 2017-01-24 20:52:43,262 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750002 with negotiated timeout 90000 for
> client /192.168.0.23:59005
>
> ----------
>
>
> I've manually checked connecting to the most relevant ports of HBase and
> ZooKeeper (using nc or telnet) and always succeeded.
>
> During a job execution, these are the type of errors that I'm getting:
>
> ----------
>
> 2017-01-24 21:10:37,187 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.29:44750
> 2017-01-24 21:10:37,193 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.29:44750
> 2017-01-24 21:10:37,198 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750003 with negotiated timeout 90000 for
> client /192.168.0.29:44750
> 2017-01-24 21:10:38,136 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750003, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:10:38,137 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.29:44750 which had sessionid 0x259d20997750003
> 2017-01-24 21:18:45,469 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49706
> 2017-01-24 21:18:45,479 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49706
> 2017-01-24 21:18:45,543 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750004 with negotiated timeout 90000 for
> client /192.168.0.19:49706
> 2017-01-24 21:18:45,684 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49708
> 2017-01-24 21:18:45,689 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49708
> 2017-01-24 21:18:45,820 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49709
> 2017-01-24 21:18:45,824 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49709
> 2017-01-24 21:18:45,825 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750005 with negotiated timeout 90000 for
> client /192.168.0.19:49708
> 2017-01-24 21:18:45,840 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750006 with negotiated timeout 90000 for
> client /192.168.0.19:49709
> 2017-01-24 21:18:45,892 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49710
> 2017-01-24 21:18:45,896 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49710
> 2017-01-24 21:18:45,965 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750007 with negotiated timeout 90000 for
> client /192.168.0.19:49710
> 2017-01-24 21:18:46,010 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49711
> 2017-01-24 21:18:46,012 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49711
> 2017-01-24 21:18:46,065 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49712
> 2017-01-24 21:18:46,067 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49712
> 2017-01-24 21:18:46,133 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750008 with negotiated timeout 90000 for
> client /192.168.0.19:49711
> 2017-01-24 21:18:46,169 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750009 with negotiated timeout 90000 for
> client /192.168.0.19:49712
> 2017-01-24 21:18:46,571 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49713
> 2017-01-24 21:18:46,578 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49713
> 2017-01-24 21:18:46,655 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d2099775000a with negotiated timeout 90000 for
> client /192.168.0.19:49713
> 2017-01-24 21:18:46,875 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49714
> 2017-01-24 21:18:46,877 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49714
> 2017-01-24 21:18:46,894 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d2099775000b with negotiated timeout 90000 for
> client /192.168.0.19:49714
> 2017-01-24 21:29:04,319 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750006, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:29:04,320 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49709 which had sessionid 0x259d20997750006
> 2017-01-24 21:29:04,344 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d2099775000a, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:29:04,345 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49713 which had sessionid 0x259d2099775000a
> 2017-01-24 21:29:04,361 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750004, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:29:04,361 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49706 which had sessionid 0x259d20997750004
> 2017-01-24 21:29:04,381 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750008, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:29:04,381 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49711 which had sessionid 0x259d20997750008
> 2017-01-24 21:29:04,392 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d2099775000b, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:29:04,393 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49714 which had sessionid 0x259d2099775000b
> 2017-01-24 21:29:04,412 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750005, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:29:04,413 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49708 which had sessionid 0x259d20997750005
> 2017-01-24 21:29:04,425 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750007, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:29:04,426 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49710 which had sessionid 0x259d20997750007
> 2017-01-24 21:29:04,445 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750009, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:29:04,446 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49712 which had sessionid 0x259d20997750009
> 2017-01-24 21:29:10,042 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49761
> 2017-01-24 21:29:10,044 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49761
> 2017-01-24 21:29:10,051 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d2099775000c with negotiated timeout 90000 for
> client /192.168.0.19:49761
> 2017-01-24 21:29:12,150 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49765
> 2017-01-24 21:29:12,152 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49765
> 2017-01-24 21:29:12,165 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d2099775000d with negotiated timeout 90000 for
> client /192.168.0.19:49765
> 2017-01-24 21:29:13,332 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49769
> 2017-01-24 21:29:13,334 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49769
> 2017-01-24 21:29:13,343 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d2099775000e with negotiated timeout 90000 for
> client /192.168.0.19:49769
> 2017-01-24 21:29:13,933 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49772
> 2017-01-24 21:29:13,935 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49772
> 2017-01-24 21:29:13,939 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d2099775000f with negotiated timeout 90000 for
> client /192.168.0.19:49772
> 2017-01-24 21:39:34,294 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d2099775000e, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:39:34,295 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49769 which had sessionid 0x259d2099775000e
> 2017-01-24 21:39:34,314 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d2099775000d, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:39:34,315 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49765 which had sessionid 0x259d2099775000d
> 2017-01-24 21:39:34,343 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d2099775000f, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:39:34,343 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49772 which had sessionid 0x259d2099775000f
> 2017-01-24 21:39:34,366 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d2099775000c, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:39:34,367 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49761 which had sessionid 0x259d2099775000c
> 2017-01-24 21:39:38,640 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49817
> 2017-01-24 21:39:38,642 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49817
> 2017-01-24 21:39:38,647 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750010 with negotiated timeout 90000 for
> client /192.168.0.19:49817
> 2017-01-24 21:39:40,386 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49821
> 2017-01-24 21:39:40,392 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49821
> 2017-01-24 21:39:40,399 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750011 with negotiated timeout 90000 for
> client /192.168.0.19:49821
> 2017-01-24 21:39:41,708 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49825
> 2017-01-24 21:39:41,712 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49825
> 2017-01-24 21:39:41,716 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750012 with negotiated timeout 90000 for
> client /192.168.0.19:49825
> 2017-01-24 21:39:42,420 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49828
> 2017-01-24 21:39:42,421 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49828
> 2017-01-24 21:39:42,426 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750013 with negotiated timeout 90000 for
> client /192.168.0.19:49828
> 2017-01-24 21:50:04,309 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750013, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:50:04,309 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49828 which had sessionid 0x259d20997750013
> 2017-01-24 21:50:04,328 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750012, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:50:04,329 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49825 which had sessionid 0x259d20997750012
> 2017-01-24 21:50:04,384 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750010, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:50:04,387 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49817 which had sessionid 0x259d20997750010
> 2017-01-24 21:50:04,396 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750011, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 21:50:04,397 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49821 which had sessionid 0x259d20997750011
> 2017-01-24 21:50:08,380 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49857
> 2017-01-24 21:50:08,382 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49857
> 2017-01-24 21:50:08,387 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750014 with negotiated timeout 90000 for
> client /192.168.0.19:49857
> 2017-01-24 21:50:09,791 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49862
> 2017-01-24 21:50:09,792 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49862
> 2017-01-24 21:50:09,799 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750015 with negotiated timeout 90000 for
> client /192.168.0.19:49862
> 2017-01-24 21:50:11,381 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49866
> 2017-01-24 21:50:11,383 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49866
> 2017-01-24 21:50:11,390 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750016 with negotiated timeout 90000 for
> client /192.168.0.19:49866
> 2017-01-24 21:50:12,189 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from /
> 192.168.0.19:49869
> 2017-01-24 21:50:12,191 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.ZooKeeperServer: Client attempting to establish new session at /
> 192.168.0.19:49869
> 2017-01-24 21:50:12,196 INFO  [CommitProcessor:2] server.ZooKeeperServer:
> Established session 0x259d20997750017 with negotiated timeout 90000 for
> client /192.168.0.19:49869
> 2017-01-24 22:00:34,306 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750015, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 22:00:34,307 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49862 which had sessionid 0x259d20997750015
> 2017-01-24 22:00:34,330 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750017, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 22:00:34,337 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49869 which had sessionid 0x259d20997750017
> 2017-01-24 22:00:34,355 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750016, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 22:00:34,356 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49866 which had sessionid 0x259d20997750016
> 2017-01-24 22:00:34,369 WARN  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: caught end of stream exception
> EndOfStreamException: Unable to read additional data from client sessionid
> 0x259d20997750014, likely client has closed socket
>         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> NIOServerCnxn.java:228)
>         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> NIOServerCnxnFactory.java:208)
>         at java.lang.Thread.run(Thread.java:745)
> 2017-01-24 22:00:34,369 INFO  [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxn: Closed socket connection for client /
> 192.168.0.19:49857 which had sessionid 0x259d20997750014
>
> ----------
>
> The RegionServer logs are clean, no warnings. I assume this is a
> ZooKeeper problem more than HBase, but I tried many different
> configurations and nothing worked. Does anyone have an idea what could
> be happening?
>
> Best,
> Hernán.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message