hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hernán Blanco <hernanblancola...@gmail.com>
Subject Re: RpcRetryingCaller error accessing HBase from MapReduce job
Date Wed, 25 Jan 2017 17:04:04 GMT
Good eye, that could be a problem. hbase-site.xml doesn't seem to be
included in the classpath... It should appear in this line of the map
log, right? 


2017-01-25 17:50:33,951 INFO [main] org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/tmp/hadoop-hdadmin/nm-local-dir/usercache/idstest/appcache/application_1485362948532_0001/container_1485362948532_0001_01_000002:/opt/hadoop-2.7.2/etc/hadoop:/opt/hadoop-2.7.2/share/hadoop/common/hadoop-nfs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2-tests.jar:/opt/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-cli-1.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-collections-3.2.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-digester-1.8.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/mockito-all-1.8.5.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/hadoop-annotations-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/hamcrest-core-1.3.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jsr305-3.0.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/junit-4.11.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-net-3.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/xz-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/gson-2.2.4.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-io-2.4.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/paranamer-2.3.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-math3-3.1.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/avro-1.7.4.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/asm-3.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/guava-11.0.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jsch-0.1.42.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/curator-recipes-2.7.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-httpclient-3.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/httpcore-4.2.5.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jettison-1.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/curator-client-2.7.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/stax-api-1.0-2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jetty-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-codec-1.4.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jets3t-0.9.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jersey-core-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/activation-1.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-lang-2.6.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/hadoop-auth-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/hadoop-hdfs-2.7.2-tests.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/hadoop-hdfs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/hadoop-hdfs-nfs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-cli-1.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-io-2.4.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/xercesImpl-2.9.1.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/asm-3.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jetty-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/htrace-core-3.1.0-incubating.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jersey-server-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-lang-2.6.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-applicationhistoryservice-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-nodemanager-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-common-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-common-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-resourcemanager-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-tests-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-applications-unmanaged-am-launcher-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-registry-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-api-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-client-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/guice-3.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-collections-3.2.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-json-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/zookeeper-3.4.6-tests.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/xz-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-io-2.4.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-client-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/asm-3.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/guava-11.0.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/zookeeper-3.4.6.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/stax-api-1.0-2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jetty-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-server-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/javax.inject-1.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-core-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/activation-1.1.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/netty-3.6.2.Final.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2-tests.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/hadoop-annotations-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/hamcrest-core-1.3.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/jersey-guice-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/junit-4.11.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/avro-1.7.4.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/asm-3.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/javax.inject-1.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/jersey-core-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/log4j-1.2.17.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.jar:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/tmp/hadoop-hdadmin/nm-local-dir/usercache/idstest/appcache/application_1485362948532_0001/container_1485362948532_0001_01_000002/job.jar:/tmp/hadoop-hdadmin/nm-local-dir/usercache/idstest/appcache/application_1485362948532_0001/container_1485362948532_0001_01_000002/hbase-common-1.2.4.jar:/tmp/hadoop-hdadmin/nm-local-dir/usercache/idstest/appcache/application_1485362948532_0001/container_1485362948532_0001_01_000002/hbase-protocol-1.2.4.jar:/tmp/hadoop-hdadmin/nm-local-dir/usercache/idstest/appcache/application_1485362948532_0001/container_1485362948532_0001_01_000002/hbase-client-1.2.4.jar


Or perhaps I'm confused with the classpaths now...

I guess I need to have the hbase-site.xml path written in the map log
under the property 

On Wed, 2017-01-25 at 07:52 -0800, Ted Yu wrote:

> From the new log snippet you posted, the hbase client tried to connect
> to 192.168.0.24 whose node is not listed in the quorum.
> 
> Can you check the classpath for the maptask ? Looks like the effective
> hbase-site.xml wasn't on the classpath.
> 
> On Wed, Jan 25, 2017 at 7:21 AM, Hernán Blanco <hernanblancolanda@gmail.com>
> wrote:
> 
> > Hello Ted,
> >
> > Thank you for your reply. Indeed, I wasn't sure if that property
> > existed, and in fact hbase-default.xml doesn't include it. I just
> > followed the advise from some webpage (I couldn't recall which) that
> > stated that you can include *any* property from the zoo.cfg native
> > config file for ZooKeeper standalone into hbase-site.xml by just adding
> > the "hbase.zookeeper.property." prefix. Even the comment in lines
> > 397-398 of
> > https://github.com/apache/hbase/blob/master/hbase-
> > common/src/main/resources/hbase-default.xml mentioned this, but I
> > probably misinterpreted it.
> >
> > I removed those incorrect properties from hbase-site.xml and reduced my
> > ZooKeeper quorum to only 3 nodes -the main and 2 others:
> >
> >
> > ...
> >   <property>
> >     <name>hbase.zookeeper.quorum</name>
> >     <value>bwcloud-fip26.rz.uni-ulm.de,bwcloud-fip27.rz.uni-ulm.de,
> > bwcloud-fip144.rz.uni-ulm.de</value>
> >   </property>
> > ...
> >
> >
> > But the problem remains. The MapReduce job keeps stuck, and same kind of
> > logs, both on HBase startup and during job execution, with an added
> > exception which is that those map tasks that are deployed on the nodes
> > that aren't running a ZooKeeper generate the following log:
> >
> >
> > ...
> > 2017-01-25 15:57:50,295 INFO [main-SendThread(bwcloud-
> > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Opening
> > socket connection to server bwcloud-fip150.rz.uni-ulm.de/192.168.0.24:2181.
> > Will not attempt to authenticate using SASL (unknown error)
> > 2017-01-25 15:57:50,296 WARN [main-SendThread(bwcloud-
> > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0
> > for server null, unexpected error, closing socket connection and attempting
> > reconnect
> > java.net.ConnectException: Connection refused
> >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >         at sun.nio.ch.SocketChannelImpl.finishConnect(
> > SocketChannelImpl.java:717)
> >         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(
> > ClientCnxnSocketNIO.java:361)
> >         at org.apache.zookeeper.ClientCnxn$SendThread.run(
> > ClientCnxn.java:1081)
> > 2017-01-25 15:57:51,400 INFO [main-SendThread(bwcloud-
> > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Opening
> > socket connection to server bwcloud-fip150.rz.uni-ulm.de/192.168.0.24:2181.
> > Will not attempt to authenticate using SASL (unknown error)
> > 2017-01-25 15:57:51,401 WARN [main-SendThread(bwcloud-
> > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Session 0x0
> > for server null, unexpected error, closing socket connection and attempting
> > reconnect
> > java.net.ConnectException: Connection refused
> >         at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> >         at sun.nio.ch.SocketChannelImpl.finishConnect(
> > SocketChannelImpl.java:717)
> >         at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(
> > ClientCnxnSocketNIO.java:361)
> >         at org.apache.zookeeper.ClientCnxn$SendThread.run(
> > ClientCnxn.java:1081)
> > ...
> >
> >
> > I actually tried this before, but my conclusion was that all nodes that
> > are running a YARN NodeManager need to have ZooKeeper running.
> >
> > Best,
> > Hernán.
> >
> >
> > On Wed, 2017-01-25 at 06:23 -0800, Ted Yu wrote:
> >
> > > bq.     <name>hbase.zookeeper.property.server.7</name>
> > >
> > > I searched 1.2 codebase but didn't find config parameter in the above
> > form.
> > > http://hbase.apache.org/book.html didn't mention it either.
> > >
> > > May I ask where you obtained such config ?
> > >
> > > For hbase.zookeeper.quorum, do you have zookeeper running on the 12
> > nodes ?
> > > Normally 3 zookeeper nodes should be enough.
> > >
> > > Cheers
> > >
> > > On Wed, Jan 25, 2017 at 2:08 AM, Hernán Blanco <
> > hernanblancolanda@gmail.com>
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I'm running HBase 1.2.4 on a distributed setup with 12 virtual machines
> > > > on the same local network. The "main" node (node26.example.com) runs
> > the
> > > > HMaster, while the other 11 machines run RegionServers. No backup
> > > > HMaster. This cluster also runs Hadoop 2.7.2 smoothly.
> > > >
> > > > Both HBase shell and accessing through the HBase client API run
> > > > properly, and even importing a TSV file into a table with
> > > > org.apache.hadoop.hbase.mapreduce.ImportTsv succeeds, remaining
> > > > registered in the YARN history.
> > > >
> > > > But the problem appears when trying to access from a MapReduce job to
> > an
> > > > HBase table (using Hadoop 2.7.2). Here a minimal code that produces the
> > > > issue by connecting and scanning an HBase table:
> > > > http://pastebin.com/pm8tbbTq. The maps hang until timeout and then
> > > > retries deploying new maps until failing, each map showing the
> > following
> > > > messages in the syslog:
> > > >
> > > > ----------
> > > >
> > > > 2017-01-24 20:04:07,904 INFO [main] org.apache.hadoop.metrics2.
> > impl.MetricsConfig:
> > > > loaded properties from hadoop-metrics2.properties
> > > > 2017-01-24 20:04:08,186 INFO [main] org.apache.hadoop.metrics2.
> > impl.MetricsSystemImpl:
> > > > Scheduled snapshot period at 10 second(s).
> > > > 2017-01-24 20:04:08,186 INFO [main] org.apache.hadoop.metrics2.
> > impl.MetricsSystemImpl:
> > > > MapTask metrics system started
> > > > 2017-01-24 20:04:08,193 INFO [main] org.apache.hadoop.mapred.
> > YarnChild:
> > > > Executing with tokens:
> > > > 2017-01-24 20:04:08,193 INFO [main] org.apache.hadoop.mapred.
> > YarnChild:
> > > > Kind: mapreduce.job, Service: job_1485182272940_0336, Ident:
> > > > (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@6f03482
> > )
> > > > 2017-01-24 20:04:08,450 INFO [main] org.apache.hadoop.mapred.
> > YarnChild:
> > > > Sleeping for 0ms before retrying again. Got null now.
> > > > 2017-01-24 20:04:08,994 INFO [main] org.apache.hadoop.mapred.
> > YarnChild:
> > > > mapreduce.cluster.local.dir for child: /tmp/hadoop-hdadmin/nm-local-
> > > > dir/usercache/idstest/appcache/application_1485182272940_0336
> > > > 2017-01-24 20:04:09,617 INFO [main] org.apache.hadoop.conf.
> > Configuration.deprecation:
> > > > session.id is deprecated. Instead, use dfs.metrics.session-id
> > > > 2017-01-24 20:04:10,968 INFO [main] org.apache.hadoop.mapreduce.
> > > > lib.output.FileOutputCommitter: File Output Committer Algorithm
> > version
> > > > is 1
> > > > 2017-01-24 20:04:11,044 INFO [main] org.apache.hadoop.mapred.Task:
> > Using
> > > > ResourceCalculatorProcessTree : [ ]
> > > > 2017-01-24 20:04:11,455 INFO [main] org.apache.hadoop.mapred.MapTask:
> > > > Processing split: hdfs://node26.example.com/
> > user/idstest/data/articles-50/
> > > > 99322.txt:0+3757
> > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask:
> > > > (EQUATOR) 0 kvi 26214396(104857584)
> > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask:
> > > > mapreduce.task.io.sort.mb: 100
> > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask:
> > > > soft limit at 83886080
> > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask:
> > > > bufstart = 0; bufvoid = 104857600
> > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTask:
> > > > kvstart = 26214396; length = 6553600
> > > > 2017-01-24 20:04:11,612 INFO [main] org.apache.hadoop.mapred.MapTask:
> > Map
> > > > output collector class = org.apache.hadoop.mapred.
> > MapTask$MapOutputBuffer
> > > > 2017-01-24 20:04:12,211 INFO [main] org.apache.hadoop.hbase.zookeeper.
> > RecoverableZooKeeper:
> > > > Process identifier=hconnection-0x650eab8 connecting to ZooKeeper
> > > > ensemble=localhost:2181
> > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:zookeeper.version=3.4.6-1569965, built on
> > 02/20/2014
> > > > 09:09 GMT
> > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:host.name=node27.example.com
> > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:java.version=1.8.0_77-Debian
> > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:java.vendor=Oracle Corporation
> > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:java.home=/usr/lib/jvm/java-8-openjdk-amd64/jre
> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:java.class.path= /* ... */
> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:java.library.path=/tmp/hadoop-hdadmin/nm-local-
> > > > dir/usercache/idstest/appcache/application_
> > 1485182272940_0336/container_
> > > > 1485182272940_0336_01_000048:/opt/hadoop-2.7.2/lib/native:/
> > > > usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/
> > > > lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:
> > /lib:/usr/lib
> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:java.io.tmpdir=/tmp/hadoop-hdadmin/nm-local-
> > > > dir/usercache/idstest/appcache/application_
> > 1485182272940_0336/container_
> > > > 1485182272940_0336_01_000048/tmp
> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:java.compiler=<NA>
> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:os.name=Linux
> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:os.arch=amd64
> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:os.version=3.16.0-4-amd64
> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:user.name=hdadmin
> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:user.home=/home/hdadmin
> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Client environment:user.dir=/tmp/hadoop-hdadmin/nm-local-dir/
> > > > usercache/idstest/appcache/application_1485182272940_
> > > > 0336/container_1485182272940_0336_01_000048
> > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper:
> > > > Initiating client connection, connectString=localhost:2181
> > > > sessionTimeout=90000 watcher=hconnection-0x650eab80x0,
> > > > quorum=localhost:2181, baseZNode=/hbase
> > > > 2017-01-24 20:04:12,322 INFO [main-SendThread(node27.example.com:2181
> > )]
> > > > org.apache.zookeeper.ClientCnxn: Opening socket connection to server
> > > > node27.example.com/192.168.0.17:2181. Will not attempt to authenticate
> > > > using SASL (unknown error)
> > > > 2017-01-24 20:04:12,323 INFO [main-SendThread(node27.example.com:2181
> > )]
> > > > org.apache.zookeeper.ClientCnxn: Socket connection established to
> > > > node27.example.com/192.168.0.17:2181, initiating session
> > > > 2017-01-24 20:04:12,331 INFO [main-SendThread(node27.example.com:2181
> > )]
> > > > org.apache.zookeeper.ClientCnxn: Session establishment complete on
> > server
> > > > node27.example.com/192.168.0.17:2181, sessionid = 0x159d07980b50189,
> > > > negotiated timeout = 90000
> > > > 2017-01-24 20:04:51,657 INFO [hconnection-0x650eab8-
> > metaLookup-shared--pool2-t1]
> > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > > > tries=10, retries=35, started=38852 ms ago, cancelled=false, msg=row
> > > > 'idstest1,,99999999999999' on table 'hbase:meta' at
> > region=hbase:meta,,1.1588230740,
> > > > hostname=node149.example.com,16020,1485261342943, seqNum=0
> > > > 2017-01-24 20:05:01,664 INFO [hconnection-0x650eab8-
> > metaLookup-shared--pool2-t1]
> > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > > > tries=11, retries=35, started=48860 ms ago, cancelled=false, msg=row
> > > > 'idstest1,,99999999999999' on table 'hbase:meta' at
> > region=hbase:meta,,1.1588230740,
> > > > hostname=node149.example.com,16020,1485261342943, seqNum=0
> > > > 2017-01-24 20:05:40,044 INFO [hconnection-0x650eab8-
> > metaLookup-shared--pool2-t2]
> > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > > > tries=10, retries=35, started=38273 ms ago, cancelled=false, msg=row
> > > > 'idstest1,,99999999999999' on table 'hbase:meta' at
> > region=hbase:meta,,1.1588230740,
> > > > hostname=node149.example.com,16020,1485261342943, seqNum=0
> > > > 2017-01-24 20:05:50,059 INFO [hconnection-0x650eab8-
> > metaLookup-shared--pool2-t2]
> > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > > > tries=11, retries=35, started=48288 ms ago, cancelled=false, msg=row
> > > > 'idstest1,,99999999999999' on table 'hbase:meta' at
> > region=hbase:meta,,1.1588230740,
> > > > hostname=node149.example.com,16020,1485261342943, seqNum=0
> > > > 2017-01-24 20:06:28,543 INFO [hconnection-0x650eab8-
> > metaLookup-shared--pool2-t3]
> > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > > > tries=10, retries=35, started=38273 ms ago, cancelled=false, msg=row
> > > > 'idstest1,,99999999999999' on table 'hbase:meta' at
> > region=hbase:meta,,1.1588230740,
> > > > hostname=node149.example.com,16020,1485261342943, seqNum=0
> > > > 2017-01-24 20:06:38,632 INFO [hconnection-0x650eab8-
> > metaLookup-shared--pool2-t3]
> > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception,
> > > > tries=11, retries=35, started=48362 ms ago, cancelled=false, msg=row
> > > > 'idstest1,,99999999999999' on table 'hbase:meta' at
> > region=hbase:meta,,1.1588230740,
> > > > hostname=node149.example.com,16020,1485261342943, seqNum=0
> > > > ...
> > > >
> > > > ----------
> > > >
> > > > My hbase-site.xml in the Master node node26.example.com (almost same
> > for
> > > > the other nodes, just referring as 0.0.0.0 to themselves):
> > > >
> > > > ----------
> > > >
> > > > <configuration>
> > > >   <property>
> > > >     <name>hbase.rootdir</name>
> > > >     <value>hdfs://node26.example.com:8020/hbase</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.dataDir</name>
> > > >     <value>/home/hdadmin/zookeeper</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.cluster.distributed</name>
> > > >     <value>true</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.quorum</name>
> > > >     <value>node26.example.com,node27.example.com,node144.example.com,
> > > > node145.example.com,node146.example.com,node147.example.com,
> > > > node148.example.com,node149.example.com,node150.example.com,
> > > > node151.example.com,node152.example.com,node153.example.com</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.0</name>
> > > >     <value>0.0.0.0:2888:3888</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.1</name>
> > > >     <value>node27.example.com:2888:3888</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.2</name>
> > > >     <value>node144.example.com:2888:3888</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.3</name>
> > > >     <value>node145.example.com:2888:3888</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.4</name>
> > > >     <value>node146.example.com:2888:3888</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.5</name>
> > > >     <value>node147.example.com:2888:3888</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.6</name>
> > > >     <value>node148.example.com:2888:3888</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.7</name>
> > > >     <value>node149.example.com:2888:3888</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.8</name>
> > > >     <value>node150.example.com:2888:3888</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.9</name>
> > > >     <value>node151.example.com:2888:3888</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.10</name>
> > > >     <value>node152.example.com:2888:3888</value>
> > > >   </property>
> > > >   <property>
> > > >     <name>hbase.zookeeper.property.server.11</name>
> > > >     <value>node153.example.com:2888:3888</value>
> > > >   </property>
> > > > </configuration>
> > > >
> > > > ----------
> > > >
> > > > After starting HBase using the start-hbase.sh script, the RegionServer
> > > > logs have no warnings or errors, but the different ZooKeeper instances
> > > > generate diverse outputs. 3 examples:
> > > >
> > > >
> > > > - The master node has an empty log:
> > > >
> > > > ----------
> > > >
> > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node26.example.com
> > > > core file size          (blocks, -c) 0
> > > > data seg size           (kbytes, -d) unlimited
> > > > scheduling priority             (-e) 0
> > > > file size               (blocks, -f) unlimited
> > > > pending signals                 (-i) 128914
> > > > max locked memory       (kbytes, -l) 64
> > > > max memory size         (kbytes, -m) unlimited
> > > > open files                      (-n) 65536
> > > > pipe size            (512 bytes, -p) 8
> > > > POSIX message queues     (bytes, -q) 819200
> > > > real-time priority              (-r) 0
> > > > stack size              (kbytes, -s) 8192
> > > > cpu time               (seconds, -t) unlimited
> > > > max user processes              (-u) 128914
> > > > virtual memory          (kbytes, -v) unlimited
> > > > file locks                      (-x) unlimited
> > > >
> > > > ----------
> > > >
> > > >
> > > > - node153.example.com has the following small ZooKeeper log:
> > > >
> > > > ----------
> > > >
> > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node153.example.com
> > > > core file size          (blocks, -c) 0
> > > > data seg size           (kbytes, -d) unlimited
> > > > scheduling priority             (-e) 0
> > > > file size               (blocks, -f) unlimited
> > > > pending signals                 (-i) 128914
> > > > max locked memory       (kbytes, -l) 64
> > > > max memory size         (kbytes, -m) unlimited
> > > > open files                      (-n) 65536
> > > > pipe size            (512 bytes, -p) 8
> > > > POSIX message queues     (bytes, -q) 819200
> > > > real-time priority              (-r) 0
> > > > stack size              (kbytes, -s) 8192
> > > > cpu time               (seconds, -t) unlimited
> > > > max user processes              (-u) 128914
> > > > virtual memory          (kbytes, -v) unlimited
> > > > file locks                      (-x) unlimited
> > > > 2017-01-24 20:52:39,816 WARN  [main] quorum.QuorumPeerConfig:
> > Non-optimial
> > > > configuration, consider an odd number of servers.
> > > > 2017-01-24 20:52:39,817 INFO  [main] quorum.QuorumPeerConfig:
> > Defaulting
> > > > to majority quorums
> > > > 2017-01-24 20:52:39,984 INFO  [main] quorum.QuorumPeerMain: Starting
> > > > quorum peer
> > > > 2017-01-24 20:52:39,994 INFO  [main] server.NIOServerCnxnFactory:
> > binding
> > > > to port 0.0.0.0/0.0.0.0:2181
> > > >
> > > > ----------
> > > >
> > > > As an extra hint, this node is not generating
> > > > the /tmp/hbase-hdadmin-zookeeper.pid, so it shows an error when
> > running
> > > > the stop-hbase.sh script.
> > > >
> > > >
> > > > - And node144.example.com (as other nodes) has the following ZooKeeper
> > > > log with "connection refused" errors:
> > > >
> > > > ----------
> > > >
> > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node144.example.com
> > > > core file size          (blocks, -c) 0
> > > > data seg size           (kbytes, -d) unlimited
> > > > scheduling priority             (-e) 0
> > > > file size               (blocks, -f) unlimited
> > > > pending signals                 (-i) 128914
> > > > max locked memory       (kbytes, -l) 64
> > > > max memory size         (kbytes, -m) unlimited
> > > > open files                      (-n) 65536
> > > > pipe size            (512 bytes, -p) 8
> > > > POSIX message queues     (bytes, -q) 819200
> > > > real-time priority              (-r) 0
> > > > stack size              (kbytes, -s) 8192
> > > > cpu time               (seconds, -t) unlimited
> > > > max user processes              (-u) 128914
> > > > virtual memory          (kbytes, -v) unlimited
> > > > file locks                      (-x) unlimited
> > > > 2017-01-24 20:52:39,473 WARN  [main] quorum.QuorumPeerConfig:
> > Non-optimial
> > > > configuration, consider an odd number of servers.
> > > > 2017-01-24 20:52:39,473 INFO  [main] quorum.QuorumPeerConfig:
> > Defaulting
> > > > to majority quorums
> > > > 2017-01-24 20:52:39,567 INFO  [main] quorum.QuorumPeerMain: Starting
> > > > quorum peer
> > > > 2017-01-24 20:52:39,581 INFO  [main] server.NIOServerCnxnFactory:
> > binding
> > > > to port 0.0.0.0/0.0.0.0:2181
> > > > 2017-01-24 20:52:39,601 INFO  [main] quorum.QuorumPeer: tickTime set to
> > > > 3000
> > > > 2017-01-24 20:52:39,601 INFO  [main] quorum.QuorumPeer:
> > minSessionTimeout
> > > > set to -1
> > > > 2017-01-24 20:52:39,601 INFO  [main] quorum.QuorumPeer:
> > maxSessionTimeout
> > > > set to 90000
> > > > 2017-01-24 20:52:39,601 INFO  [main] quorum.QuorumPeer: initLimit set
> > to 10
> > > > 2017-01-24 20:52:39,613 INFO  [main] persistence.FileSnap: Reading
> > > > snapshot /home/hdadmin/zookeeper2/version-2/snapshot.1d0000023f
> > > > 2017-01-24 20:52:39,714 INFO  [Thread-2] quorum.QuorumCnxManager: My
> > > > election bind port: node144.example.com/192.168.0.19:3888
> > > > 2017-01-24 20:52:39,729 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > quorum.QuorumPeer: LOOKING
> > > > 2017-01-24 20:52:39,731 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > quorum.FastLeaderElection: New election. My id =  2, proposed
> > > > zxid=0x1e0000258d
> > > > 2017-01-24 20:52:39,740 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 2
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 2
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,740 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 0
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 0
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,742 WARN  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Cannot open channel to 3 at election address
> > > > node145.example.com/192.168.0.18:3888
> > > > java.net.ConnectException: Connection refused
> > > >         at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > >         at java.net.AbstractPlainSocketImpl.doConnect(
> > > > AbstractPlainSocketImpl.java:350)
> > > >         at java.net.AbstractPlainSocketImpl.connectToAddress(
> > > > AbstractPlainSocketImpl.java:206)
> > > >         at java.net.AbstractPlainSocketImpl.connect(
> > > > AbstractPlainSocketImpl.java:188)
> > > >         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > > >         at java.net.Socket.connect(Socket.java:589)
> > > >         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> > > > connectOne(QuorumCnxManager.java:368)
> > > >         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> > > > toSend(QuorumCnxManager.java:341)
> > > >         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> > > > Messenger$WorkerSender.process(FastLeaderElection.java:449)
> > > >         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> > > > Messenger$WorkerSender.run(FastLeaderElection.java:430)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 20:52:39,744 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 1
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,745 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 1
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,745 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 2
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 0
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,746 INFO  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping
> > the
> > > > connection: (4, 2)
> > > > 2017-01-24 20:52:39,747 INFO  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping
> > the
> > > > connection: (5, 2)
> > > > 2017-01-24 20:52:39,747 INFO  [node144.example.com/192.168.0.19:3888]
> > > > quorum.QuorumCnxManager: Received connection request /
> > 192.168.0.21:40064
> > > > 2017-01-24 20:52:39,747 INFO  [node144.example.com/192.168.0.19:3888]
> > > > quorum.QuorumCnxManager: Received connection request /
> > 192.168.0.22:54362
> > > > 2017-01-24 20:52:39,747 INFO  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping
> > the
> > > > connection: (6, 2)
> > > > 2017-01-24 20:52:39,748 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 4
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,748 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 5
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,748 INFO  [node144.example.com/192.168.0.19:3888]
> > > > quorum.QuorumCnxManager: Received connection request /
> > 192.168.0.23:53410
> > > > 2017-01-24 20:52:39,749 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 5
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,749 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 0
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,749 INFO  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping
> > the
> > > > connection: (7, 2)
> > > > 2017-01-24 20:52:39,749 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 4
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,749 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 6
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,750 INFO  [node144.example.com/192.168.0.19:3888]
> > > > quorum.QuorumCnxManager: Received connection request /
> > 192.168.0.25:33085
> > > > 2017-01-24 20:52:39,750 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 6
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,751 WARN  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Cannot open channel to 8 at election address
> > > > node150.example.com/192.168.0.24:3888
> > > > java.net.ConnectException: Connection refused
> > > >         at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > >         at java.net.AbstractPlainSocketImpl.doConnect(
> > > > AbstractPlainSocketImpl.java:350)
> > > >         at java.net.AbstractPlainSocketImpl.connectToAddress(
> > > > AbstractPlainSocketImpl.java:206)
> > > >         at java.net.AbstractPlainSocketImpl.connect(
> > > > AbstractPlainSocketImpl.java:188)
> > > >         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > > >         at java.net.Socket.connect(Socket.java:589)
> > > >         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> > > > connectOne(QuorumCnxManager.java:368)
> > > >         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> > > > toSend(QuorumCnxManager.java:341)
> > > >         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> > > > Messenger$WorkerSender.process(FastLeaderElection.java:449)
> > > >         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> > > > Messenger$WorkerSender.run(FastLeaderElection.java:430)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 20:52:39,751 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 7
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,752 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 7
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,753 INFO  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping
> > the
> > > > connection: (9, 2)
> > > > 2017-01-24 20:52:39,753 INFO  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping
> > the
> > > > connection: (10, 2)
> > > > 2017-01-24 20:52:39,753 INFO  [node144.example.com/192.168.0.19:3888]
> > > > quorum.QuorumCnxManager: Received connection request /
> > 192.168.0.26:35997
> > > > 2017-01-24 20:52:39,754 INFO  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping
> > the
> > > > connection: (11, 2)
> > > > 2017-01-24 20:52:39,755 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 2
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,755 INFO  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping
> > the
> > > > connection: (3, 2)
> > > > 2017-01-24 20:52:39,756 WARN  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Cannot open channel to 8 at election address
> > > > node150.example.com/192.168.0.24:3888
> > > > java.net.ConnectException: Connection refused
> > > >         at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > >         at java.net.AbstractPlainSocketImpl.doConnect(
> > > > AbstractPlainSocketImpl.java:350)
> > > >         at java.net.AbstractPlainSocketImpl.connectToAddress(
> > > > AbstractPlainSocketImpl.java:206)
> > > >         at java.net.AbstractPlainSocketImpl.connect(
> > > > AbstractPlainSocketImpl.java:188)
> > > >         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > > >         at java.net.Socket.connect(Socket.java:589)
> > > >         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> > > > connectOne(QuorumCnxManager.java:368)
> > > >         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> > > > toSend(QuorumCnxManager.java:341)
> > > >         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> > > > Messenger$WorkerSender.process(FastLeaderElection.java:449)
> > > >         at org.apache.zookeeper.server.quorum.FastLeaderElection$
> > > > Messenger$WorkerSender.run(FastLeaderElection.java:430)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 20:52:39,757 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 9
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,757 INFO  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping
> > the
> > > > connection: (10, 2)
> > > > 2017-01-24 20:52:39,757 INFO  [WorkerSender[myid=2]]
> > > > quorum.QuorumCnxManager: Have smaller server identifier, so dropping
> > the
> > > > connection: (11, 2)
> > > > 2017-01-24 20:52:39,758 INFO  [node144.example.com/192.168.0.19:3888]
> > > > quorum.QuorumCnxManager: Received connection request /
> > 192.168.0.27:34404
> > > > 2017-01-24 20:52:39,759 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 9
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,768 INFO  [node144.example.com/192.168.0.19:3888]
> > > > quorum.QuorumCnxManager: Received connection request /
> > 192.168.0.18:59032
> > > > 2017-01-24 20:52:39,769 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 11
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,769 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 3
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 3
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,776 INFO  [node144.example.com/192.168.0.19:3888]
> > > > quorum.QuorumCnxManager: Received connection request /
> > 192.168.0.27:34405
> > > > 2017-01-24 20:52:39,776 WARN  [RecvWorker:11] quorum.QuorumCnxManager:
> > > > Connection broken for id 11, my id = 2, error =
> > > > java.io.EOFException
> > > >         at java.io.DataInputStream.readInt(DataInputStream.java:392)
> > > >         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> > > > RecvWorker.run(QuorumCnxManager.java:765)
> > > > 2017-01-24 20:52:39,777 WARN  [RecvWorker:11] quorum.QuorumCnxManager:
> > > > Interrupting SendWorker
> > > > 2017-01-24 20:52:39,777 WARN  [SendWorker:11] quorum.QuorumCnxManager:
> > > > Interrupted while waiting for message on queue
> > > > java.lang.InterruptedException
> > > >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> > > > ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.
> > > > java:2014)
> > > >         at java.util.concurrent.locks.AbstractQueuedSynchronizer$
> > > > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088)
> > > >         at java.util.concurrent.ArrayBlockingQueue.poll(
> > > > ArrayBlockingQueue.java:418)
> > > >         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> > > > pollSendQueue(QuorumCnxManager.java:849)
> > > >         at org.apache.zookeeper.server.quorum.QuorumCnxManager.
> > > > access$500(QuorumCnxManager.java:64)
> > > >         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> > > > SendWorker.run(QuorumCnxManager.java:685)
> > > > 2017-01-24 20:52:39,777 INFO  [node144.example.com/192.168.0.19:3888]
> > > > quorum.QuorumCnxManager: Received connection request /
> > 192.168.0.28:57974
> > > > 2017-01-24 20:52:39,778 WARN  [SendWorker:11] quorum.QuorumCnxManager:
> > > > Send worker leaving thread
> > > > 2017-01-24 20:52:39,778 INFO  [node144.example.com/192.168.0.19:3888]
> > > > quorum.QuorumCnxManager: Received connection request /
> > 192.168.0.28:57975
> > > > 2017-01-24 20:52:39,778 WARN  [RecvWorker:10] quorum.QuorumCnxManager:
> > > > Connection broken for id 10, my id = 2, error =
> > > > java.io.EOFException
> > > >         at java.io.DataInputStream.readInt(DataInputStream.java:392)
> > > >         at org.apache.zookeeper.server.quorum.QuorumCnxManager$
> > > > RecvWorker.run(QuorumCnxManager.java:765)
> > > > 2017-01-24 20:52:39,778 WARN  [RecvWorker:10] quorum.QuorumCnxManager:
> > > > Interrupting SendWorker
> > > > 2017-01-24 20:52:39,778 WARN  [SendWorker:10] quorum.QuorumCnxManager:
> > > > Exception when using channel: for id 10 my id = 2 error =
> > > > java.net.SocketException: Broken pipe
> > > > 2017-01-24 20:52:39,778 WARN  [SendWorker:10] quorum.QuorumCnxManager:
> > > > Send worker leaving thread
> > > > 2017-01-24 20:52:39,779 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 3
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,782 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 10
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 10
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,788 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 11
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,789 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 10
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,815 INFO  [node144.example.com/192.168.0.19:3888]
> > > > quorum.QuorumCnxManager: Received connection request /
> > 192.168.0.24:56503
> > > > 2017-01-24 20:52:39,829 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 8
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state), 8
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:39,829 INFO  [WorkerReceiver[myid=2]]
> > > > quorum.FastLeaderElection: Notification: 1 (message format version), 11
> > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state), 8
> > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state)
> > > > 2017-01-24 20:52:40,030 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > quorum.QuorumPeer: FOLLOWING
> > > > 2017-01-24 20:52:40,035 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > quorum.Learner: TCP NoDelay set to: true
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:zookeeper.version=
> > 3.4.6-1569965,
> > > > built on 02/20/2014 09:09 GMT
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:host.name=node144.
> > example.com
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:java.version=1.8.
> > 0_77-Debian
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:java.vendor=Oracle
> > Corporation
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:java.home=/usr/
> > > > lib/jvm/java-8-openjdk-amd64/jre
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:java.class.path= /* ... */
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:java.library.path=
> > > > /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/
> > > > lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:
> > /lib:/usr/lib
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:java.io.tmpdir=/tmp
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:java.compiler=<NA>
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:os.name=Linux
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:os.arch=amd64
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:os.version=3.16.0-4-amd64
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:user.name=hdadmin
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:user.home=/home/hdadmin
> > > > 2017-01-24 20:52:40,040 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Server environment:user.dir=/opt/hbase-1.2.4
> > > > 2017-01-24 20:52:40,042 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > server.ZooKeeperServer: Created server with tickTime 3000
> > minSessionTimeout
> > > > 6000 maxSessionTimeout 90000 datadir /home/hdadmin/zookeeper2/
> > version-2
> > > > snapdir /home/hdadmin/zookeeper2/version-2
> > > > 2017-01-24 20:52:40,043 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > quorum.Learner: FOLLOWING - LEADER ELECTION TOOK - 311
> > > > 2017-01-24 20:52:40,046 WARN  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > quorum.Learner: Unexpected exception, tries=0, connecting to
> > > > node153.example.com/192.168.0.27:2888
> > > > java.net.ConnectException: Connection refused
> > > >         at java.net.PlainSocketImpl.socketConnect(Native Method)
> > > >         at java.net.AbstractPlainSocketImpl.doConnect(
> > > > AbstractPlainSocketImpl.java:350)
> > > >         at java.net.AbstractPlainSocketImpl.connectToAddress(
> > > > AbstractPlainSocketImpl.java:206)
> > > >         at java.net.AbstractPlainSocketImpl.connect(
> > > > AbstractPlainSocketImpl.java:188)
> > > >         at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> > > >         at java.net.Socket.connect(Socket.java:589)
> > > >         at org.apache.zookeeper.server.quorum.Learner.
> > > > connectToLeader(Learner.java:225)
> > > >         at org.apache.zookeeper.server.quorum.Follower.followLeader(
> > > > Follower.java:71)
> > > >         at org.apache.zookeeper.server.quorum.QuorumPeer.run(
> > > > QuorumPeer.java:786)
> > > > 2017-01-24 20:52:41,073 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > quorum.Learner: Getting a diff from the leader 0x1e0000258d
> > > > 2017-01-24 20:52:41,116 INFO  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > persistence.FileTxnSnapLog: Snapshotting: 0x1e0000258d to
> > > > /home/hdadmin/zookeeper2/version-2/snapshot.1e0000258d
> > > > 2017-01-24 20:52:42,408 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.29:44336
> > > > 2017-01-24 20:52:42,418 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.29:44336
> > > > 2017-01-24 20:52:42,421 WARN  [QuorumPeer[myid=2]/0:0:0:0:0:
> > 0:0:0:2181]
> > > > quorum.Learner: Got zxid 0x1f00000001 expected 0x1
> > > > 2017-01-24 20:52:42,421 INFO  [SyncThread:2] persistence.FileTxnLog:
> > > > Creating new log file: log.1f00000001
> > > > 2017-01-24 20:52:42,440 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750000 with negotiated timeout 90000 for
> > > > client /192.168.0.29:44336
> > > > 2017-01-24 20:52:42,734 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.22:43377
> > > > 2017-01-24 20:52:42,737 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.22:43377
> > > > 2017-01-24 20:52:42,742 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750001 with negotiated timeout 90000 for
> > > > client /192.168.0.22:43377
> > > > 2017-01-24 20:52:43,257 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.23:59005
> > > > 2017-01-24 20:52:43,258 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.23:59005
> > > > 2017-01-24 20:52:43,262 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750002 with negotiated timeout 90000 for
> > > > client /192.168.0.23:59005
> > > >
> > > > ----------
> > > >
> > > >
> > > > I've manually checked connecting to the most relevant ports of HBase
> > and
> > > > ZooKeeper (using nc or telnet) and always succeeded.
> > > >
> > > > During a job execution, these are the type of errors that I'm getting:
> > > >
> > > > ----------
> > > >
> > > > 2017-01-24 21:10:37,187 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.29:44750
> > > > 2017-01-24 21:10:37,193 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.29:44750
> > > > 2017-01-24 21:10:37,198 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750003 with negotiated timeout 90000 for
> > > > client /192.168.0.29:44750
> > > > 2017-01-24 21:10:38,136 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750003, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:10:38,137 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.29:44750 which had sessionid 0x259d20997750003
> > > > 2017-01-24 21:18:45,469 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49706
> > > > 2017-01-24 21:18:45,479 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49706
> > > > 2017-01-24 21:18:45,543 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750004 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49706
> > > > 2017-01-24 21:18:45,684 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49708
> > > > 2017-01-24 21:18:45,689 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49708
> > > > 2017-01-24 21:18:45,820 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49709
> > > > 2017-01-24 21:18:45,824 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49709
> > > > 2017-01-24 21:18:45,825 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750005 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49708
> > > > 2017-01-24 21:18:45,840 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750006 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49709
> > > > 2017-01-24 21:18:45,892 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49710
> > > > 2017-01-24 21:18:45,896 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49710
> > > > 2017-01-24 21:18:45,965 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750007 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49710
> > > > 2017-01-24 21:18:46,010 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49711
> > > > 2017-01-24 21:18:46,012 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49711
> > > > 2017-01-24 21:18:46,065 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49712
> > > > 2017-01-24 21:18:46,067 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49712
> > > > 2017-01-24 21:18:46,133 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750008 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49711
> > > > 2017-01-24 21:18:46,169 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750009 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49712
> > > > 2017-01-24 21:18:46,571 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49713
> > > > 2017-01-24 21:18:46,578 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49713
> > > > 2017-01-24 21:18:46,655 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d2099775000a with negotiated timeout 90000 for
> > > > client /192.168.0.19:49713
> > > > 2017-01-24 21:18:46,875 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49714
> > > > 2017-01-24 21:18:46,877 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49714
> > > > 2017-01-24 21:18:46,894 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d2099775000b with negotiated timeout 90000 for
> > > > client /192.168.0.19:49714
> > > > 2017-01-24 21:29:04,319 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750006, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:29:04,320 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49709 which had sessionid 0x259d20997750006
> > > > 2017-01-24 21:29:04,344 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d2099775000a, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:29:04,345 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49713 which had sessionid 0x259d2099775000a
> > > > 2017-01-24 21:29:04,361 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750004, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:29:04,361 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49706 which had sessionid 0x259d20997750004
> > > > 2017-01-24 21:29:04,381 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750008, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:29:04,381 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49711 which had sessionid 0x259d20997750008
> > > > 2017-01-24 21:29:04,392 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d2099775000b, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:29:04,393 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49714 which had sessionid 0x259d2099775000b
> > > > 2017-01-24 21:29:04,412 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750005, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:29:04,413 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49708 which had sessionid 0x259d20997750005
> > > > 2017-01-24 21:29:04,425 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750007, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:29:04,426 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49710 which had sessionid 0x259d20997750007
> > > > 2017-01-24 21:29:04,445 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750009, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:29:04,446 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49712 which had sessionid 0x259d20997750009
> > > > 2017-01-24 21:29:10,042 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49761
> > > > 2017-01-24 21:29:10,044 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49761
> > > > 2017-01-24 21:29:10,051 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d2099775000c with negotiated timeout 90000 for
> > > > client /192.168.0.19:49761
> > > > 2017-01-24 21:29:12,150 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49765
> > > > 2017-01-24 21:29:12,152 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49765
> > > > 2017-01-24 21:29:12,165 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d2099775000d with negotiated timeout 90000 for
> > > > client /192.168.0.19:49765
> > > > 2017-01-24 21:29:13,332 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49769
> > > > 2017-01-24 21:29:13,334 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49769
> > > > 2017-01-24 21:29:13,343 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d2099775000e with negotiated timeout 90000 for
> > > > client /192.168.0.19:49769
> > > > 2017-01-24 21:29:13,933 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49772
> > > > 2017-01-24 21:29:13,935 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49772
> > > > 2017-01-24 21:29:13,939 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d2099775000f with negotiated timeout 90000 for
> > > > client /192.168.0.19:49772
> > > > 2017-01-24 21:39:34,294 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d2099775000e, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:39:34,295 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49769 which had sessionid 0x259d2099775000e
> > > > 2017-01-24 21:39:34,314 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d2099775000d, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:39:34,315 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49765 which had sessionid 0x259d2099775000d
> > > > 2017-01-24 21:39:34,343 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d2099775000f, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:39:34,343 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49772 which had sessionid 0x259d2099775000f
> > > > 2017-01-24 21:39:34,366 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d2099775000c, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:39:34,367 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49761 which had sessionid 0x259d2099775000c
> > > > 2017-01-24 21:39:38,640 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49817
> > > > 2017-01-24 21:39:38,642 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49817
> > > > 2017-01-24 21:39:38,647 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750010 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49817
> > > > 2017-01-24 21:39:40,386 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49821
> > > > 2017-01-24 21:39:40,392 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49821
> > > > 2017-01-24 21:39:40,399 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750011 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49821
> > > > 2017-01-24 21:39:41,708 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49825
> > > > 2017-01-24 21:39:41,712 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49825
> > > > 2017-01-24 21:39:41,716 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750012 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49825
> > > > 2017-01-24 21:39:42,420 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49828
> > > > 2017-01-24 21:39:42,421 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49828
> > > > 2017-01-24 21:39:42,426 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750013 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49828
> > > > 2017-01-24 21:50:04,309 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750013, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:50:04,309 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49828 which had sessionid 0x259d20997750013
> > > > 2017-01-24 21:50:04,328 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750012, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:50:04,329 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49825 which had sessionid 0x259d20997750012
> > > > 2017-01-24 21:50:04,384 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750010, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:50:04,387 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49817 which had sessionid 0x259d20997750010
> > > > 2017-01-24 21:50:04,396 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750011, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 21:50:04,397 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49821 which had sessionid 0x259d20997750011
> > > > 2017-01-24 21:50:08,380 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49857
> > > > 2017-01-24 21:50:08,382 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49857
> > > > 2017-01-24 21:50:08,387 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750014 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49857
> > > > 2017-01-24 21:50:09,791 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49862
> > > > 2017-01-24 21:50:09,792 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49862
> > > > 2017-01-24 21:50:09,799 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750015 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49862
> > > > 2017-01-24 21:50:11,381 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49866
> > > > 2017-01-24 21:50:11,383 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49866
> > > > 2017-01-24 21:50:11,390 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750016 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49866
> > > > 2017-01-24 21:50:12,189 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxnFactory: Accepted socket connection from /
> > > > 192.168.0.19:49869
> > > > 2017-01-24 21:50:12,191 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.ZooKeeperServer: Client attempting to establish new session at /
> > > > 192.168.0.19:49869
> > > > 2017-01-24 21:50:12,196 INFO  [CommitProcessor:2]
> > server.ZooKeeperServer:
> > > > Established session 0x259d20997750017 with negotiated timeout 90000 for
> > > > client /192.168.0.19:49869
> > > > 2017-01-24 22:00:34,306 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750015, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 22:00:34,307 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49862 which had sessionid 0x259d20997750015
> > > > 2017-01-24 22:00:34,330 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750017, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 22:00:34,337 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49869 which had sessionid 0x259d20997750017
> > > > 2017-01-24 22:00:34,355 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750016, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 22:00:34,356 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49866 which had sessionid 0x259d20997750016
> > > > 2017-01-24 22:00:34,369 WARN  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: caught end of stream exception
> > > > EndOfStreamException: Unable to read additional data from client
> > sessionid
> > > > 0x259d20997750014, likely client has closed socket
> > > >         at org.apache.zookeeper.server.NIOServerCnxn.doIO(
> > > > NIOServerCnxn.java:228)
> > > >         at org.apache.zookeeper.server.NIOServerCnxnFactory.run(
> > > > NIOServerCnxnFactory.java:208)
> > > >         at java.lang.Thread.run(Thread.java:745)
> > > > 2017-01-24 22:00:34,369 INFO  [NIOServerCxn.Factory:0.0.0.0/
> > 0.0.0.0:2181]
> > > > server.NIOServerCnxn: Closed socket connection for client /
> > > > 192.168.0.19:49857 which had sessionid 0x259d20997750014
> > > >
> > > > ----------
> > > >
> > > > The RegionServer logs are clean, no warnings. I assume this is a
> > > > ZooKeeper problem more than HBase, but I tried many different
> > > > configurations and nothing worked. Does anyone have an idea what could
> > > > be happening?
> > > >
> > > > Best,
> > > > Hernán.
> > > >
> >
> >
> >



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message