Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A73DC200C09 for ; Wed, 25 Jan 2017 18:04:29 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id A5A25160B5A; Wed, 25 Jan 2017 17:04:29 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 28F6C160B3D for ; Wed, 25 Jan 2017 18:04:27 +0100 (CET) Received: (qmail 37363 invoked by uid 500); 25 Jan 2017 17:04:25 -0000 Mailing-List: contact user-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hbase.apache.org Delivered-To: mailing list user@hbase.apache.org Received: (qmail 37351 invoked by uid 99); 25 Jan 2017 17:04:25 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 25 Jan 2017 17:04:25 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D0BBEC0FC9 for ; Wed, 25 Jan 2017 17:04:24 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.091 X-Spam-Level: *** X-Spam-Status: No, score=3.091 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, HTML_TAG_BALANCE_BODY=0.712, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id feGHlGSdAEod for ; Wed, 25 Jan 2017 17:04:14 +0000 (UTC) Received: from mail-wm0-f42.google.com (mail-wm0-f42.google.com [74.125.82.42]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id B84F25FC06 for ; Wed, 25 Jan 2017 17:04:13 +0000 (UTC) Received: by mail-wm0-f42.google.com with SMTP id c206so41598558wme.0 for ; Wed, 25 Jan 2017 09:04:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:subject:from:to:date:in-reply-to:references:mime-version; bh=bitWVKZqvhNYq6PqnvuqqxnITrTdqxzNvHQxLkGRGaQ=; b=SxEFgsJ2+Fw3UFdytWXA7nu6xmeljptHyN1AH0fg83NSeuxIQFAdB7a9Xmk2TgXkXg +aPqMA/apEsThql+RQ9qwTCi0dZz/CIogVzv2K+QK/EQS/aVK2DeyT/W1A22J+KXayNB ir3HkZq0c+W0lKnr3Ge3kjcPqYZahDZlwN4AE2t1Wy/HH3RTfAavF5yVVj8ULY31bkKU t8nkZe8Y5BHDHLAFfr3klmf8VflU3rpfu3I1V6tjga69ra4hVdgXfMEr/7Bsl7psO8Ng 6hM1jFVxXvqt1pHLNb8oIRnLu8BimSZw4BRFbEsIzky5txAe0R8/yOXvvpaVbiDY+LeC yyKA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:date:in-reply-to :references:mime-version; bh=bitWVKZqvhNYq6PqnvuqqxnITrTdqxzNvHQxLkGRGaQ=; b=JLJ0Yl1qVhmt23f+SruEsZ70TpAr/z5nFcQu7ZPY8UGI8BGBvYyz/pet086txAUKO1 W4XC12wlBNDI8KqsnbyNxUtPx8OlyJfHkynyhArujncfv2VsBs//DX7F6U3l1kP2mumM +0hq94/g2g3YarbBH9ifqa+g3w/nuxVCG7rEsbL+bWwRIFAvdSLlFtT8TaXznqUDglHp QdJlWz8pzmo+U4LHjzYFw6p8oUH8g6OhLBQ/G0CZVhf1HVt6o/50Gv8db1np2Zx2IUHi C0Vk5EKYTyDUgr+x+v7Vt+E9lIuSytfsYSnqaVlyw0ZmXREpJHEIaIx/J3tvaBnOfyoe zaqA== X-Gm-Message-State: AIkVDXJKJL8aZXz+chT9MWuhmUwgXxjjTy6ES+Luhh7e+v439U+yLHu4S4UVZ3neum+zuA== X-Received: by 10.223.142.208 with SMTP id q74mr34163813wrb.101.1485363845824; Wed, 25 Jan 2017 09:04:05 -0800 (PST) Received: from hblanco (db71-194.informatik.uni-ulm.de. [134.60.71.194]) by smtp.googlemail.com with ESMTPSA id q1sm1591181wmd.6.2017.01.25.09.04.04 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 25 Jan 2017 09:04:05 -0800 (PST) Message-ID: <1485363844.1810.56.camel@gmail.com> Subject: Re: RpcRetryingCaller error accessing HBase from MapReduce job From: =?ISO-8859-1?Q?Hern=E1n?= Blanco To: user@hbase.apache.org Date: Wed, 25 Jan 2017 18:04:04 +0100 In-Reply-To: References: <1485338912.1810.19.camel@gmail.com> <1485357702.1810.41.camel@gmail.com> Content-Type: multipart/alternative; boundary="=-qe22pRSkFwiKX8Ea3+yG" X-Mailer: Evolution 3.12.9-1+b1 Mime-Version: 1.0 archived-at: Wed, 25 Jan 2017 17:04:29 -0000 --=-qe22pRSkFwiKX8Ea3+yG Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Good eye, that could be a problem. hbase-site.xml doesn't seem to be included in the classpath... It should appear in this line of the map log, right?=20 2017-01-25 17:50:33,951 INFO [main] org.apache.zookeeper.ZooKeeper: Client = environment:java.class.path=3D/tmp/hadoop-hdadmin/nm-local-dir/usercache/id= stest/appcache/application_1485362948532_0001/container_1485362948532_0001_= 01_000002:/opt/hadoop-2.7.2/etc/hadoop:/opt/hadoop-2.7.2/share/hadoop/commo= n/hadoop-nfs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/hadoop-common-= 2.7.2-tests.jar:/opt/hadoop-2.7.2/share/hadoop/common/hadoop-common-2.7.2.j= ar:/opt/hadoop-2.7.2/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-= M15.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/slf4j-api-1.7.10.jar:/opt= /hadoop-2.7.2/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/opt/hado= op-2.7.2/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/opt/hadoop-2.7.2/= share/hadoop/common/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.7.2/sha= re/hadoop/common/lib/commons-cli-1.2.jar:/opt/hadoop-2.7.2/share/hadoop/com= mon/lib/commons-collections-3.2.2.jar:/opt/hadoop-2.7.2/share/hadoop/common= /lib/commons-digester-1.8.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/moc= kito-all-1.8.5.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/hadoop-annotat= ions-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/hamcrest-core-1.3.= jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jersey-json-1.9.jar:/opt/hado= op-2.7.2/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.7.2/= share/hadoop/common/lib/jackson-xc-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoo= p/common/lib/jsr305-3.0.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jet= ty-util-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-compre= ss-1.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/junit-4.11.jar:/opt/= hadoop-2.7.2/share/hadoop/common/lib/commons-configuration-1.6.jar:/opt/had= oop-2.7.2/share/hadoop/common/lib/commons-net-3.1.jar:/opt/hadoop-2.7.2/sha= re/hadoop/common/lib/jaxb-api-2.2.2.jar:/opt/hadoop-2.7.2/share/hadoop/comm= on/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/= xz-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/api-util-1.0.0-M20.jar= :/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar= :/opt/hadoop-2.7.2/share/hadoop/common/lib/gson-2.2.4.jar:/opt/hadoop-2.7.2= /share/hadoop/common/lib/commons-io-2.4.jar:/opt/hadoop-2.7.2/share/hadoop/= common/lib/slf4j-log4j12-1.7.10.jar:/opt/hadoop-2.7.2/share/hadoop/common/l= ib/paranamer-2.3.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-math= 3-3.1.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/xmlenc-0.52.jar:/opt/= hadoop-2.7.2/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/opt/hadoo= p-2.7.2/share/hadoop/common/lib/servlet-api-2.5.jar:/opt/hadoop-2.7.2/share= /hadoop/common/lib/avro-1.7.4.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib= /asm-3.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/guava-11.0.2.jar:/op= t/hadoop-2.7.2/share/hadoop/common/lib/zookeeper-3.4.6.jar:/opt/hadoop-2.7.= 2/share/hadoop/common/lib/curator-framework-2.7.1.jar:/opt/hadoop-2.7.2/sha= re/hadoop/common/lib/jackson-jaxrs-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoo= p/common/lib/jsch-0.1.42.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/cura= tor-recipes-2.7.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/commons-htt= pclient-3.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/httpcore-4.2.5.ja= r:/opt/hadoop-2.7.2/share/hadoop/common/lib/jettison-1.1.jar:/opt/hadoop-2.= 7.2/share/hadoop/common/lib/jsp-api-2.1.jar:/opt/hadoop-2.7.2/share/hadoop/= common/lib/curator-client-2.7.1.jar:/opt/hadoop-2.7.2/share/hadoop/common/l= ib/stax-api-1.0-2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jetty-6.1.2= 6.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/httpclient-4.2.5.jar:/opt/h= adoop-2.7.2/share/hadoop/common/lib/htrace-core-3.1.0-incubating.jar:/opt/h= adoop-2.7.2/share/hadoop/common/lib/jersey-server-1.9.jar:/opt/hadoop-2.7.2= /share/hadoop/common/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.7.2/shar= e/hadoop/common/lib/commons-codec-1.4.jar:/opt/hadoop-2.7.2/share/hadoop/co= mmon/lib/api-asn1-api-1.0.0-M20.jar:/opt/hadoop-2.7.2/share/hadoop/common/l= ib/jets3t-0.9.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jersey-core-1= .9.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/log4j-1.2.17.jar:/opt/hado= op-2.7.2/share/hadoop/common/lib/activation-1.1.jar:/opt/hadoop-2.7.2/share= /hadoop/common/lib/commons-lang-2.6.jar:/opt/hadoop-2.7.2/share/hadoop/comm= on/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/ha= doop-auth-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/netty-3.6.2.F= inal.jar:/opt/hadoop-2.7.2/share/hadoop/common/lib/jaxb-impl-2.2.3-1.jar:/o= pt/hadoop-2.7.2/share/hadoop/hdfs/hadoop-hdfs-2.7.2-tests.jar:/opt/hadoop-2= .7.2/share/hadoop/hdfs/hadoop-hdfs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop= /hdfs/hadoop-hdfs-nfs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jac= kson-mapper-asl-1.9.13.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-= cli-1.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/leveldbjni-all-1.8.jar:= /opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jsr305-3.0.0.jar:/opt/hadoop-2.7.2/= share/hadoop/hdfs/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/= hdfs/lib/commons-logging-1.1.3.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/= commons-io-2.4.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/xercesImpl-2.9.1= .jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/xmlenc-0.52.jar:/opt/hadoop-2.= 7.2/share/hadoop/hdfs/lib/commons-daemon-1.0.13.jar:/opt/hadoop-2.7.2/share= /hadoop/hdfs/lib/servlet-api-2.5.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/li= b/asm-3.2.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/guava-11.0.2.jar:/opt= /hadoop-2.7.2/share/hadoop/hdfs/lib/netty-all-4.0.23.Final.jar:/opt/hadoop-= 2.7.2/share/hadoop/hdfs/lib/xml-apis-1.3.04.jar:/opt/hadoop-2.7.2/share/had= oop/hdfs/lib/jetty-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/htrac= e-core-3.1.0-incubating.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jersey-= server-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/jackson-core-asl-1.9= .13.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/commons-codec-1.4.jar:/opt/= hadoop-2.7.2/share/hadoop/hdfs/lib/jersey-core-1.9.jar:/opt/hadoop-2.7.2/sh= are/hadoop/hdfs/lib/log4j-1.2.17.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/li= b/commons-lang-2.6.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/protobuf-jav= a-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/hdfs/lib/netty-3.6.2.Final.jar:/= opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-sharedcachemanager-2.= 7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-applicationh= istoryservice-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-ser= ver-nodemanager-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-c= ommon-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-comm= on-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-resourc= emanager-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-server-t= ests-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-applications= -unmanaged-am-launcher-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop= -yarn-registry-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-ap= plications-distributedshell-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/h= adoop-yarn-api-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-se= rver-web-proxy-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/hadoop-yarn-cl= ient-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/guice-3.0.jar:/opt/h= adoop-2.7.2/share/hadoop/yarn/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop= -2.7.2/share/hadoop/yarn/lib/commons-cli-1.2.jar:/opt/hadoop-2.7.2/share/ha= doop/yarn/lib/commons-collections-3.2.2.jar:/opt/hadoop-2.7.2/share/hadoop/= yarn/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jer= sey-json-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jackson-xc-1.9.13.= jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-guice-1.9.jar:/opt/hadoo= p-2.7.2/share/hadoop/yarn/lib/jsr305-3.0.0.jar:/opt/hadoop-2.7.2/share/hado= op/yarn/lib/jetty-util-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/c= ommons-compress-1.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/zookeeper= -3.4.6-tests.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jaxb-api-2.2.2.jar= :/opt/hadoop-2.7.2/share/hadoop/yarn/lib/commons-logging-1.1.3.jar:/opt/had= oop-2.7.2/share/hadoop/yarn/lib/xz-1.0.jar:/opt/hadoop-2.7.2/share/hadoop/y= arn/lib/commons-io-2.4.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-c= lient-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/servlet-api-2.5.jar:/= opt/hadoop-2.7.2/share/hadoop/yarn/lib/asm-3.2.jar:/opt/hadoop-2.7.2/share/= hadoop/yarn/lib/guava-11.0.2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/zo= okeeper-3.4.6.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jackson-jaxrs-1.9= .13.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jettison-1.1.jar:/opt/hadoo= p-2.7.2/share/hadoop/yarn/lib/aopalliance-1.0.jar:/opt/hadoop-2.7.2/share/h= adoop/yarn/lib/stax-api-1.0-2.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/j= etty-6.1.26.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey-server-1.9.j= ar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jackson-core-asl-1.9.13.jar:/opt= /hadoop-2.7.2/share/hadoop/yarn/lib/commons-codec-1.4.jar:/opt/hadoop-2.7.2= /share/hadoop/yarn/lib/guice-servlet-3.0.jar:/opt/hadoop-2.7.2/share/hadoop= /yarn/lib/javax.inject-1.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jersey= -core-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/log4j-1.2.17.jar:/opt= /hadoop-2.7.2/share/hadoop/yarn/lib/activation-1.1.jar:/opt/hadoop-2.7.2/sh= are/hadoop/yarn/lib/commons-lang-2.6.jar:/opt/hadoop-2.7.2/share/hadoop/yar= n/lib/protobuf-java-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/netty= -3.6.2.Final.jar:/opt/hadoop-2.7.2/share/hadoop/yarn/lib/jaxb-impl-2.2.3-1.= jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-common= -2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-client= -app-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-mapreduce-cl= ient-hs-plugins-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoop-m= apreduce-client-hs-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/hadoo= p-mapreduce-client-core-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/= hadoop-mapreduce-client-jobclient-2.7.2-tests.jar:/opt/hadoop-2.7.2/share/h= adoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar:/opt/hadoop-2.7.2/share= /hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.7.2.jar:/opt/hadoop-2= .7.2/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.7.2.jar:/opt/= hadoop-2.7.2/share/hadoop/mapreduce/lib/guice-3.0.jar:/opt/hadoop-2.7.2/sha= re/hadoop/mapreduce/lib/jackson-mapper-asl-1.9.13.jar:/opt/hadoop-2.7.2/sha= re/hadoop/mapreduce/lib/leveldbjni-all-1.8.jar:/opt/hadoop-2.7.2/share/hado= op/mapreduce/lib/hadoop-annotations-2.7.2.jar:/opt/hadoop-2.7.2/share/hadoo= p/mapreduce/lib/hamcrest-core-1.3.jar:/opt/hadoop-2.7.2/share/hadoop/mapred= uce/lib/snappy-java-1.0.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/li= b/jersey-guice-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/commons= -compress-1.4.1.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/junit-4.11= .jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/xz-1.0.jar:/opt/hadoop-2.= 7.2/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/opt/hadoop-2.7.2/share/h= adoop/mapreduce/lib/paranamer-2.3.jar:/opt/hadoop-2.7.2/share/hadoop/mapred= uce/lib/avro-1.7.4.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/asm-3.2= .jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/opt/= hadoop-2.7.2/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/opt/hadoop-2= .7.2/share/hadoop/mapreduce/lib/jackson-core-asl-1.9.13.jar:/opt/hadoop-2.7= .2/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/opt/hadoop-2.7.2/share= /hadoop/mapreduce/lib/javax.inject-1.jar:/opt/hadoop-2.7.2/share/hadoop/map= reduce/lib/jersey-core-1.9.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib= /log4j-1.2.17.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/protobuf-jav= a-2.5.0.jar:/opt/hadoop-2.7.2/share/hadoop/mapreduce/lib/netty-3.6.2.Final.= jar:job.jar/job.jar:job.jar/classes/:job.jar/lib/*:/tmp/hadoop-hdadmin/nm-l= ocal-dir/usercache/idstest/appcache/application_1485362948532_0001/containe= r_1485362948532_0001_01_000002/job.jar:/tmp/hadoop-hdadmin/nm-local-dir/use= rcache/idstest/appcache/application_1485362948532_0001/container_1485362948= 532_0001_01_000002/hbase-common-1.2.4.jar:/tmp/hadoop-hdadmin/nm-local-dir/= usercache/idstest/appcache/application_1485362948532_0001/container_1485362= 948532_0001_01_000002/hbase-protocol-1.2.4.jar:/tmp/hadoop-hdadmin/nm-local= -dir/usercache/idstest/appcache/application_1485362948532_0001/container_14= 85362948532_0001_01_000002/hbase-client-1.2.4.jar Or perhaps I'm confused with the classpaths now... I guess I need to have the hbase-site.xml path written in the map log under the property=20 On Wed, 2017-01-25 at 07:52 -0800, Ted Yu wrote: > From the new log snippet you posted, the hbase client tried to connect > to 192.168.0.24 whose node is not listed in the quorum. >=20 > Can you check the classpath for the maptask ? Looks like the effective > hbase-site.xml wasn't on the classpath. >=20 > On Wed, Jan 25, 2017 at 7:21 AM, Hern=C3=A1n Blanco > wrote: >=20 > > Hello Ted, > > > > Thank you for your reply. Indeed, I wasn't sure if that property > > existed, and in fact hbase-default.xml doesn't include it. I just > > followed the advise from some webpage (I couldn't recall which) that > > stated that you can include *any* property from the zoo.cfg native > > config file for ZooKeeper standalone into hbase-site.xml by just adding > > the "hbase.zookeeper.property." prefix. Even the comment in lines > > 397-398 of > > https://github.com/apache/hbase/blob/master/hbase- > > common/src/main/resources/hbase-default.xml mentioned this, but I > > probably misinterpreted it. > > > > I removed those incorrect properties from hbase-site.xml and reduced my > > ZooKeeper quorum to only 3 nodes -the main and 2 others: > > > > > > ... > > > > hbase.zookeeper.quorum > > bwcloud-fip26.rz.uni-ulm.de,bwcloud-fip27.rz.uni-ulm.de, > > bwcloud-fip144.rz.uni-ulm.de > > > > ... > > > > > > But the problem remains. The MapReduce job keeps stuck, and same kind o= f > > logs, both on HBase startup and during job execution, with an added > > exception which is that those map tasks that are deployed on the nodes > > that aren't running a ZooKeeper generate the following log: > > > > > > ... > > 2017-01-25 15:57:50,295 INFO [main-SendThread(bwcloud- > > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Opening > > socket connection to server bwcloud-fip150.rz.uni-ulm.de/192.168.0.24:2= 181. > > Will not attempt to authenticate using SASL (unknown error) > > 2017-01-25 15:57:50,296 WARN [main-SendThread(bwcloud- > > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Session 0x= 0 > > for server null, unexpected error, closing socket connection and attemp= ting > > reconnect > > java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at sun.nio.ch.SocketChannelImpl.finishConnect( > > SocketChannelImpl.java:717) > > at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport( > > ClientCnxnSocketNIO.java:361) > > at org.apache.zookeeper.ClientCnxn$SendThread.run( > > ClientCnxn.java:1081) > > 2017-01-25 15:57:51,400 INFO [main-SendThread(bwcloud- > > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Opening > > socket connection to server bwcloud-fip150.rz.uni-ulm.de/192.168.0.24:2= 181. > > Will not attempt to authenticate using SASL (unknown error) > > 2017-01-25 15:57:51,401 WARN [main-SendThread(bwcloud- > > fip150.rz.uni-ulm.de:2181)] org.apache.zookeeper.ClientCnxn: Session 0x= 0 > > for server null, unexpected error, closing socket connection and attemp= ting > > reconnect > > java.net.ConnectException: Connection refused > > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > > at sun.nio.ch.SocketChannelImpl.finishConnect( > > SocketChannelImpl.java:717) > > at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport( > > ClientCnxnSocketNIO.java:361) > > at org.apache.zookeeper.ClientCnxn$SendThread.run( > > ClientCnxn.java:1081) > > ... > > > > > > I actually tried this before, but my conclusion was that all nodes that > > are running a YARN NodeManager need to have ZooKeeper running. > > > > Best, > > Hern=C3=A1n. > > > > > > On Wed, 2017-01-25 at 06:23 -0800, Ted Yu wrote: > > > > > bq. hbase.zookeeper.property.server.7 > > > > > > I searched 1.2 codebase but didn't find config parameter in the above > > form. > > > http://hbase.apache.org/book.html didn't mention it either. > > > > > > May I ask where you obtained such config ? > > > > > > For hbase.zookeeper.quorum, do you have zookeeper running on the 12 > > nodes ? > > > Normally 3 zookeeper nodes should be enough. > > > > > > Cheers > > > > > > On Wed, Jan 25, 2017 at 2:08 AM, Hern=C3=A1n Blanco < > > hernanblancolanda@gmail.com> > > > wrote: > > > > > > > Hi all, > > > > > > > > I'm running HBase 1.2.4 on a distributed setup with 12 virtual mach= ines > > > > on the same local network. The "main" node (node26.example.com) run= s > > the > > > > HMaster, while the other 11 machines run RegionServers. No backup > > > > HMaster. This cluster also runs Hadoop 2.7.2 smoothly. > > > > > > > > Both HBase shell and accessing through the HBase client API run > > > > properly, and even importing a TSV file into a table with > > > > org.apache.hadoop.hbase.mapreduce.ImportTsv succeeds, remaining > > > > registered in the YARN history. > > > > > > > > But the problem appears when trying to access from a MapReduce job = to > > an > > > > HBase table (using Hadoop 2.7.2). Here a minimal code that produces= the > > > > issue by connecting and scanning an HBase table: > > > > http://pastebin.com/pm8tbbTq. The maps hang until timeout and then > > > > retries deploying new maps until failing, each map showing the > > following > > > > messages in the syslog: > > > > > > > > ---------- > > > > > > > > 2017-01-24 20:04:07,904 INFO [main] org.apache.hadoop.metrics2. > > impl.MetricsConfig: > > > > loaded properties from hadoop-metrics2.properties > > > > 2017-01-24 20:04:08,186 INFO [main] org.apache.hadoop.metrics2. > > impl.MetricsSystemImpl: > > > > Scheduled snapshot period at 10 second(s). > > > > 2017-01-24 20:04:08,186 INFO [main] org.apache.hadoop.metrics2. > > impl.MetricsSystemImpl: > > > > MapTask metrics system started > > > > 2017-01-24 20:04:08,193 INFO [main] org.apache.hadoop.mapred. > > YarnChild: > > > > Executing with tokens: > > > > 2017-01-24 20:04:08,193 INFO [main] org.apache.hadoop.mapred. > > YarnChild: > > > > Kind: mapreduce.job, Service: job_1485182272940_0336, Ident: > > > > (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@6f03= 482 > > ) > > > > 2017-01-24 20:04:08,450 INFO [main] org.apache.hadoop.mapred. > > YarnChild: > > > > Sleeping for 0ms before retrying again. Got null now. > > > > 2017-01-24 20:04:08,994 INFO [main] org.apache.hadoop.mapred. > > YarnChild: > > > > mapreduce.cluster.local.dir for child: /tmp/hadoop-hdadmin/nm-local= - > > > > dir/usercache/idstest/appcache/application_1485182272940_0336 > > > > 2017-01-24 20:04:09,617 INFO [main] org.apache.hadoop.conf. > > Configuration.deprecation: > > > > session.id is deprecated. Instead, use dfs.metrics.session-id > > > > 2017-01-24 20:04:10,968 INFO [main] org.apache.hadoop.mapreduce. > > > > lib.output.FileOutputCommitter: File Output Committer Algorithm > > version > > > > is 1 > > > > 2017-01-24 20:04:11,044 INFO [main] org.apache.hadoop.mapred.Task: > > Using > > > > ResourceCalculatorProcessTree : [ ] > > > > 2017-01-24 20:04:11,455 INFO [main] org.apache.hadoop.mapred.MapTas= k: > > > > Processing split: hdfs://node26.example.com/ > > user/idstest/data/articles-50/ > > > > 99322.txt:0+3757 > > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTas= k: > > > > (EQUATOR) 0 kvi 26214396(104857584) > > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTas= k: > > > > mapreduce.task.io.sort.mb: 100 > > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTas= k: > > > > soft limit at 83886080 > > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTas= k: > > > > bufstart =3D 0; bufvoid =3D 104857600 > > > > 2017-01-24 20:04:11,596 INFO [main] org.apache.hadoop.mapred.MapTas= k: > > > > kvstart =3D 26214396; length =3D 6553600 > > > > 2017-01-24 20:04:11,612 INFO [main] org.apache.hadoop.mapred.MapTas= k: > > Map > > > > output collector class =3D org.apache.hadoop.mapred. > > MapTask$MapOutputBuffer > > > > 2017-01-24 20:04:12,211 INFO [main] org.apache.hadoop.hbase.zookeep= er. > > RecoverableZooKeeper: > > > > Process identifier=3Dhconnection-0x650eab8 connecting to ZooKeeper > > > > ensemble=3Dlocalhost:2181 > > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:zookeeper.version=3D3.4.6-1569965, built on > > 02/20/2014 > > > > 09:09 GMT > > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:host.name=3Dnode27.example.com > > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.version=3D1.8.0_77-Debian > > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.vendor=3DOracle Corporation > > > > 2017-01-24 20:04:12,234 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.home=3D/usr/lib/jvm/java-8-openjdk-amd64/jr= e > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.class.path=3D /* ... */ > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.library.path=3D/tmp/hadoop-hdadmin/nm-local= - > > > > dir/usercache/idstest/appcache/application_ > > 1485182272940_0336/container_ > > > > 1485182272940_0336_01_000048:/opt/hadoop-2.7.2/lib/native:/ > > > > usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/ > > > > lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni: > > /lib:/usr/lib > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.io.tmpdir=3D/tmp/hadoop-hdadmin/nm-local- > > > > dir/usercache/idstest/appcache/application_ > > 1485182272940_0336/container_ > > > > 1485182272940_0336_01_000048/tmp > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:java.compiler=3D > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:os.name=3DLinux > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:os.arch=3Damd64 > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:os.version=3D3.16.0-4-amd64 > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:user.name=3Dhdadmin > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:user.home=3D/home/hdadmin > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Client environment:user.dir=3D/tmp/hadoop-hdadmin/nm-local-dir/ > > > > usercache/idstest/appcache/application_1485182272940_ > > > > 0336/container_1485182272940_0336_01_000048 > > > > 2017-01-24 20:04:12,235 INFO [main] org.apache.zookeeper.ZooKeeper: > > > > Initiating client connection, connectString=3Dlocalhost:2181 > > > > sessionTimeout=3D90000 watcher=3Dhconnection-0x650eab80x0, > > > > quorum=3Dlocalhost:2181, baseZNode=3D/hbase > > > > 2017-01-24 20:04:12,322 INFO [main-SendThread(node27.example.com:21= 81 > > )] > > > > org.apache.zookeeper.ClientCnxn: Opening socket connection to serve= r > > > > node27.example.com/192.168.0.17:2181. Will not attempt to authentic= ate > > > > using SASL (unknown error) > > > > 2017-01-24 20:04:12,323 INFO [main-SendThread(node27.example.com:21= 81 > > )] > > > > org.apache.zookeeper.ClientCnxn: Socket connection established to > > > > node27.example.com/192.168.0.17:2181, initiating session > > > > 2017-01-24 20:04:12,331 INFO [main-SendThread(node27.example.com:21= 81 > > )] > > > > org.apache.zookeeper.ClientCnxn: Session establishment complete on > > server > > > > node27.example.com/192.168.0.17:2181, sessionid =3D 0x159d07980b501= 89, > > > > negotiated timeout =3D 90000 > > > > 2017-01-24 20:04:51,657 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t1] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=3D10, retries=3D35, started=3D38852 ms ago, cancelled=3Dfalse= , msg=3Drow > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=3Dhbase:meta,,1.1588230740, > > > > hostname=3Dnode149.example.com,16020,1485261342943, seqNum=3D0 > > > > 2017-01-24 20:05:01,664 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t1] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=3D11, retries=3D35, started=3D48860 ms ago, cancelled=3Dfalse= , msg=3Drow > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=3Dhbase:meta,,1.1588230740, > > > > hostname=3Dnode149.example.com,16020,1485261342943, seqNum=3D0 > > > > 2017-01-24 20:05:40,044 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t2] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=3D10, retries=3D35, started=3D38273 ms ago, cancelled=3Dfalse= , msg=3Drow > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=3Dhbase:meta,,1.1588230740, > > > > hostname=3Dnode149.example.com,16020,1485261342943, seqNum=3D0 > > > > 2017-01-24 20:05:50,059 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t2] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=3D11, retries=3D35, started=3D48288 ms ago, cancelled=3Dfalse= , msg=3Drow > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=3Dhbase:meta,,1.1588230740, > > > > hostname=3Dnode149.example.com,16020,1485261342943, seqNum=3D0 > > > > 2017-01-24 20:06:28,543 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t3] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=3D10, retries=3D35, started=3D38273 ms ago, cancelled=3Dfalse= , msg=3Drow > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=3Dhbase:meta,,1.1588230740, > > > > hostname=3Dnode149.example.com,16020,1485261342943, seqNum=3D0 > > > > 2017-01-24 20:06:38,632 INFO [hconnection-0x650eab8- > > metaLookup-shared--pool2-t3] > > > > org.apache.hadoop.hbase.client.RpcRetryingCaller: Call exception, > > > > tries=3D11, retries=3D35, started=3D48362 ms ago, cancelled=3Dfalse= , msg=3Drow > > > > 'idstest1,,99999999999999' on table 'hbase:meta' at > > region=3Dhbase:meta,,1.1588230740, > > > > hostname=3Dnode149.example.com,16020,1485261342943, seqNum=3D0 > > > > ... > > > > > > > > ---------- > > > > > > > > My hbase-site.xml in the Master node node26.example.com (almost sam= e > > for > > > > the other nodes, just referring as 0.0.0.0 to themselves): > > > > > > > > ---------- > > > > > > > > > > > > > > > > hbase.rootdir > > > > hdfs://node26.example.com:8020/hbase > > > > > > > > > > > > hbase.zookeeper.property.dataDir > > > > /home/hdadmin/zookeeper > > > > > > > > > > > > hbase.cluster.distributed > > > > true > > > > > > > > > > > > hbase.zookeeper.quorum > > > > node26.example.com,node27.example.com,node144.example.co= m, > > > > node145.example.com,node146.example.com,node147.example.com, > > > > node148.example.com,node149.example.com,node150.example.com, > > > > node151.example.com,node152.example.com,node153.example.com > > > > > > > > > > > > hbase.zookeeper.property.server.0 > > > > 0.0.0.0:2888:3888 > > > > > > > > > > > > hbase.zookeeper.property.server.1 > > > > node27.example.com:2888:3888 > > > > > > > > > > > > hbase.zookeeper.property.server.2 > > > > node144.example.com:2888:3888 > > > > > > > > > > > > hbase.zookeeper.property.server.3 > > > > node145.example.com:2888:3888 > > > > > > > > > > > > hbase.zookeeper.property.server.4 > > > > node146.example.com:2888:3888 > > > > > > > > > > > > hbase.zookeeper.property.server.5 > > > > node147.example.com:2888:3888 > > > > > > > > > > > > hbase.zookeeper.property.server.6 > > > > node148.example.com:2888:3888 > > > > > > > > > > > > hbase.zookeeper.property.server.7 > > > > node149.example.com:2888:3888 > > > > > > > > > > > > hbase.zookeeper.property.server.8 > > > > node150.example.com:2888:3888 > > > > > > > > > > > > hbase.zookeeper.property.server.9 > > > > node151.example.com:2888:3888 > > > > > > > > > > > > hbase.zookeeper.property.server.10 > > > > node152.example.com:2888:3888 > > > > > > > > > > > > hbase.zookeeper.property.server.11 > > > > node153.example.com:2888:3888 > > > > > > > > > > > > > > > > ---------- > > > > > > > > After starting HBase using the start-hbase.sh script, the RegionSer= ver > > > > logs have no warnings or errors, but the different ZooKeeper instan= ces > > > > generate diverse outputs. 3 examples: > > > > > > > > > > > > - The master node has an empty log: > > > > > > > > ---------- > > > > > > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node26.example.c= om > > > > core file size (blocks, -c) 0 > > > > data seg size (kbytes, -d) unlimited > > > > scheduling priority (-e) 0 > > > > file size (blocks, -f) unlimited > > > > pending signals (-i) 128914 > > > > max locked memory (kbytes, -l) 64 > > > > max memory size (kbytes, -m) unlimited > > > > open files (-n) 65536 > > > > pipe size (512 bytes, -p) 8 > > > > POSIX message queues (bytes, -q) 819200 > > > > real-time priority (-r) 0 > > > > stack size (kbytes, -s) 8192 > > > > cpu time (seconds, -t) unlimited > > > > max user processes (-u) 128914 > > > > virtual memory (kbytes, -v) unlimited > > > > file locks (-x) unlimited > > > > > > > > ---------- > > > > > > > > > > > > - node153.example.com has the following small ZooKeeper log: > > > > > > > > ---------- > > > > > > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node153.example.= com > > > > core file size (blocks, -c) 0 > > > > data seg size (kbytes, -d) unlimited > > > > scheduling priority (-e) 0 > > > > file size (blocks, -f) unlimited > > > > pending signals (-i) 128914 > > > > max locked memory (kbytes, -l) 64 > > > > max memory size (kbytes, -m) unlimited > > > > open files (-n) 65536 > > > > pipe size (512 bytes, -p) 8 > > > > POSIX message queues (bytes, -q) 819200 > > > > real-time priority (-r) 0 > > > > stack size (kbytes, -s) 8192 > > > > cpu time (seconds, -t) unlimited > > > > max user processes (-u) 128914 > > > > virtual memory (kbytes, -v) unlimited > > > > file locks (-x) unlimited > > > > 2017-01-24 20:52:39,816 WARN [main] quorum.QuorumPeerConfig: > > Non-optimial > > > > configuration, consider an odd number of servers. > > > > 2017-01-24 20:52:39,817 INFO [main] quorum.QuorumPeerConfig: > > Defaulting > > > > to majority quorums > > > > 2017-01-24 20:52:39,984 INFO [main] quorum.QuorumPeerMain: Startin= g > > > > quorum peer > > > > 2017-01-24 20:52:39,994 INFO [main] server.NIOServerCnxnFactory: > > binding > > > > to port 0.0.0.0/0.0.0.0:2181 > > > > > > > > ---------- > > > > > > > > As an extra hint, this node is not generating > > > > the /tmp/hbase-hdadmin-zookeeper.pid, so it shows an error when > > running > > > > the stop-hbase.sh script. > > > > > > > > > > > > - And node144.example.com (as other nodes) has the following ZooKee= per > > > > log with "connection refused" errors: > > > > > > > > ---------- > > > > > > > > Tue Jan 24 20:52:38 CET 2017 Starting zookeeper on node144.example.= com > > > > core file size (blocks, -c) 0 > > > > data seg size (kbytes, -d) unlimited > > > > scheduling priority (-e) 0 > > > > file size (blocks, -f) unlimited > > > > pending signals (-i) 128914 > > > > max locked memory (kbytes, -l) 64 > > > > max memory size (kbytes, -m) unlimited > > > > open files (-n) 65536 > > > > pipe size (512 bytes, -p) 8 > > > > POSIX message queues (bytes, -q) 819200 > > > > real-time priority (-r) 0 > > > > stack size (kbytes, -s) 8192 > > > > cpu time (seconds, -t) unlimited > > > > max user processes (-u) 128914 > > > > virtual memory (kbytes, -v) unlimited > > > > file locks (-x) unlimited > > > > 2017-01-24 20:52:39,473 WARN [main] quorum.QuorumPeerConfig: > > Non-optimial > > > > configuration, consider an odd number of servers. > > > > 2017-01-24 20:52:39,473 INFO [main] quorum.QuorumPeerConfig: > > Defaulting > > > > to majority quorums > > > > 2017-01-24 20:52:39,567 INFO [main] quorum.QuorumPeerMain: Startin= g > > > > quorum peer > > > > 2017-01-24 20:52:39,581 INFO [main] server.NIOServerCnxnFactory: > > binding > > > > to port 0.0.0.0/0.0.0.0:2181 > > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: tickTime se= t to > > > > 3000 > > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: > > minSessionTimeout > > > > set to -1 > > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: > > maxSessionTimeout > > > > set to 90000 > > > > 2017-01-24 20:52:39,601 INFO [main] quorum.QuorumPeer: initLimit s= et > > to 10 > > > > 2017-01-24 20:52:39,613 INFO [main] persistence.FileSnap: Reading > > > > snapshot /home/hdadmin/zookeeper2/version-2/snapshot.1d0000023f > > > > 2017-01-24 20:52:39,714 INFO [Thread-2] quorum.QuorumCnxManager: M= y > > > > election bind port: node144.example.com/192.168.0.19:3888 > > > > 2017-01-24 20:52:39,729 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.QuorumPeer: LOOKING > > > > 2017-01-24 20:52:39,731 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.FastLeaderElection: New election. My id =3D 2, proposed > > > > zxid=3D0x1e0000258d > > > > 2017-01-24 20:52:39,740 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 2 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state)= , 2 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,740 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 0 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state)= , 0 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,742 WARN [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Cannot open channel to 3 at election addre= ss > > > > node145.example.com/192.168.0.18:3888 > > > > java.net.ConnectException: Connection refused > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > > AbstractPlainSocketImpl.java:350) > > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > > AbstractPlainSocketImpl.java:206) > > > > at java.net.AbstractPlainSocketImpl.connect( > > > > AbstractPlainSocketImpl.java:188) > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:39= 2) > > > > at java.net.Socket.connect(Socket.java:589) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > connectOne(QuorumCnxManager.java:368) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > toSend(QuorumCnxManager.java:341) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.process(FastLeaderElection.java:449) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.run(FastLeaderElection.java:430) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 20:52:39,744 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 1 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,745 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 1 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,745 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 2 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state)= , 0 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,746 INFO [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so droppin= g > > the > > > > connection: (4, 2) > > > > 2017-01-24 20:52:39,747 INFO [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so droppin= g > > the > > > > connection: (5, 2) > > > > 2017-01-24 20:52:39,747 INFO [node144.example.com/192.168.0.19:388= 8] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.21:40064 > > > > 2017-01-24 20:52:39,747 INFO [node144.example.com/192.168.0.19:388= 8] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.22:54362 > > > > 2017-01-24 20:52:39,747 INFO [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so droppin= g > > the > > > > connection: (6, 2) > > > > 2017-01-24 20:52:39,748 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 4 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,748 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 5 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,748 INFO [node144.example.com/192.168.0.19:388= 8] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.23:53410 > > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 5 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 0 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,749 INFO [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so droppin= g > > the > > > > connection: (7, 2) > > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 4 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,749 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 6 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,750 INFO [node144.example.com/192.168.0.19:388= 8] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.25:33085 > > > > 2017-01-24 20:52:39,750 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 6 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,751 WARN [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Cannot open channel to 8 at election addre= ss > > > > node150.example.com/192.168.0.24:3888 > > > > java.net.ConnectException: Connection refused > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > > AbstractPlainSocketImpl.java:350) > > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > > AbstractPlainSocketImpl.java:206) > > > > at java.net.AbstractPlainSocketImpl.connect( > > > > AbstractPlainSocketImpl.java:188) > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:39= 2) > > > > at java.net.Socket.connect(Socket.java:589) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > connectOne(QuorumCnxManager.java:368) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > toSend(QuorumCnxManager.java:341) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.process(FastLeaderElection.java:449) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.run(FastLeaderElection.java:430) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 20:52:39,751 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 7 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,752 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 7 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,753 INFO [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so droppin= g > > the > > > > connection: (9, 2) > > > > 2017-01-24 20:52:39,753 INFO [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so droppin= g > > the > > > > connection: (10, 2) > > > > 2017-01-24 20:52:39,753 INFO [node144.example.com/192.168.0.19:388= 8] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.26:35997 > > > > 2017-01-24 20:52:39,754 INFO [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so droppin= g > > the > > > > connection: (11, 2) > > > > 2017-01-24 20:52:39,755 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 2 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,755 INFO [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so droppin= g > > the > > > > connection: (3, 2) > > > > 2017-01-24 20:52:39,756 WARN [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Cannot open channel to 8 at election addre= ss > > > > node150.example.com/192.168.0.24:3888 > > > > java.net.ConnectException: Connection refused > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > > AbstractPlainSocketImpl.java:350) > > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > > AbstractPlainSocketImpl.java:206) > > > > at java.net.AbstractPlainSocketImpl.connect( > > > > AbstractPlainSocketImpl.java:188) > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:39= 2) > > > > at java.net.Socket.connect(Socket.java:589) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > connectOne(QuorumCnxManager.java:368) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > toSend(QuorumCnxManager.java:341) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.process(FastLeaderElection.java:449) > > > > at org.apache.zookeeper.server.quorum.FastLeaderElection$ > > > > Messenger$WorkerSender.run(FastLeaderElection.java:430) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 20:52:39,757 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 9 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,757 INFO [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so droppin= g > > the > > > > connection: (10, 2) > > > > 2017-01-24 20:52:39,757 INFO [WorkerSender[myid=3D2]] > > > > quorum.QuorumCnxManager: Have smaller server identifier, so droppin= g > > the > > > > connection: (11, 2) > > > > 2017-01-24 20:52:39,758 INFO [node144.example.com/192.168.0.19:388= 8] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.27:34404 > > > > 2017-01-24 20:52:39,759 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 9 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,768 INFO [node144.example.com/192.168.0.19:388= 8] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.18:59032 > > > > 2017-01-24 20:52:39,769 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 11 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,769 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 3 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state)= , 3 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,776 INFO [node144.example.com/192.168.0.19:388= 8] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.27:34405 > > > > 2017-01-24 20:52:39,776 WARN [RecvWorker:11] quorum.QuorumCnxManag= er: > > > > Connection broken for id 11, my id =3D 2, error =3D > > > > java.io.EOFException > > > > at java.io.DataInputStream.readInt(DataInputStream.java:392= ) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager$ > > > > RecvWorker.run(QuorumCnxManager.java:765) > > > > 2017-01-24 20:52:39,777 WARN [RecvWorker:11] quorum.QuorumCnxManag= er: > > > > Interrupting SendWorker > > > > 2017-01-24 20:52:39,777 WARN [SendWorker:11] quorum.QuorumCnxManag= er: > > > > Interrupted while waiting for message on queue > > > > java.lang.InterruptedException > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > > ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer= . > > > > java:2014) > > > > at java.util.concurrent.locks.AbstractQueuedSynchronizer$ > > > > ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2088) > > > > at java.util.concurrent.ArrayBlockingQueue.poll( > > > > ArrayBlockingQueue.java:418) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > pollSendQueue(QuorumCnxManager.java:849) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager. > > > > access$500(QuorumCnxManager.java:64) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager$ > > > > SendWorker.run(QuorumCnxManager.java:685) > > > > 2017-01-24 20:52:39,777 INFO [node144.example.com/192.168.0.19:388= 8] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.28:57974 > > > > 2017-01-24 20:52:39,778 WARN [SendWorker:11] quorum.QuorumCnxManag= er: > > > > Send worker leaving thread > > > > 2017-01-24 20:52:39,778 INFO [node144.example.com/192.168.0.19:388= 8] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.28:57975 > > > > 2017-01-24 20:52:39,778 WARN [RecvWorker:10] quorum.QuorumCnxManag= er: > > > > Connection broken for id 10, my id =3D 2, error =3D > > > > java.io.EOFException > > > > at java.io.DataInputStream.readInt(DataInputStream.java:392= ) > > > > at org.apache.zookeeper.server.quorum.QuorumCnxManager$ > > > > RecvWorker.run(QuorumCnxManager.java:765) > > > > 2017-01-24 20:52:39,778 WARN [RecvWorker:10] quorum.QuorumCnxManag= er: > > > > Interrupting SendWorker > > > > 2017-01-24 20:52:39,778 WARN [SendWorker:10] quorum.QuorumCnxManag= er: > > > > Exception when using channel: for id 10 my id =3D 2 error =3D > > > > java.net.SocketException: Broken pipe > > > > 2017-01-24 20:52:39,778 WARN [SendWorker:10] quorum.QuorumCnxManag= er: > > > > Send worker leaving thread > > > > 2017-01-24 20:52:39,779 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 3 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,782 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 10 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state)= , 10 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,788 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 11 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,789 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 10 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,815 INFO [node144.example.com/192.168.0.19:388= 8] > > > > quorum.QuorumCnxManager: Received connection request / > > 192.168.0.24:56503 > > > > 2017-01-24 20:52:39,829 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 8 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x1 (n.round), LOOKING (n.state)= , 8 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:39,829 INFO [WorkerReceiver[myid=3D2]] > > > > quorum.FastLeaderElection: Notification: 1 (message format version)= , 11 > > > > (n.leader), 0x1e0000258d (n.zxid), 0x9 (n.round), LOOKING (n.state)= , 8 > > > > (n.sid), 0x1e (n.peerEpoch) LOOKING (my state) > > > > 2017-01-24 20:52:40,030 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.QuorumPeer: FOLLOWING > > > > 2017-01-24 20:52:40,035 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.Learner: TCP NoDelay set to: true > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:zookeeper.version=3D > > 3.4.6-1569965, > > > > built on 02/20/2014 09:09 GMT > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:host.name=3Dnode144. > > example.com > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.version=3D1.8. > > 0_77-Debian > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.vendor=3DOracle > > Corporation > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.home=3D/usr/ > > > > lib/jvm/java-8-openjdk-amd64/jre > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.class.path=3D /* ..= . */ > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.library.path=3D > > > > /usr/java/packages/lib/amd64:/usr/lib/x86_64-linux-gnu/jni:/ > > > > lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni: > > /lib:/usr/lib > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.io.tmpdir=3D/tmp > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:java.compiler=3D > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:os.name=3DLinux > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:os.arch=3Damd64 > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:os.version=3D3.16.0-4-am= d64 > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:user.name=3Dhdadmin > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:user.home=3D/home/hdadmi= n > > > > 2017-01-24 20:52:40,040 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Server environment:user.dir=3D/opt/hbase-1.= 2.4 > > > > 2017-01-24 20:52:40,042 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > server.ZooKeeperServer: Created server with tickTime 3000 > > minSessionTimeout > > > > 6000 maxSessionTimeout 90000 datadir /home/hdadmin/zookeeper2/ > > version-2 > > > > snapdir /home/hdadmin/zookeeper2/version-2 > > > > 2017-01-24 20:52:40,043 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.Learner: FOLLOWING - LEADER ELECTION TOOK - 311 > > > > 2017-01-24 20:52:40,046 WARN [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.Learner: Unexpected exception, tries=3D0, connecting to > > > > node153.example.com/192.168.0.27:2888 > > > > java.net.ConnectException: Connection refused > > > > at java.net.PlainSocketImpl.socketConnect(Native Method) > > > > at java.net.AbstractPlainSocketImpl.doConnect( > > > > AbstractPlainSocketImpl.java:350) > > > > at java.net.AbstractPlainSocketImpl.connectToAddress( > > > > AbstractPlainSocketImpl.java:206) > > > > at java.net.AbstractPlainSocketImpl.connect( > > > > AbstractPlainSocketImpl.java:188) > > > > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:39= 2) > > > > at java.net.Socket.connect(Socket.java:589) > > > > at org.apache.zookeeper.server.quorum.Learner. > > > > connectToLeader(Learner.java:225) > > > > at org.apache.zookeeper.server.quorum.Follower.followLeader= ( > > > > Follower.java:71) > > > > at org.apache.zookeeper.server.quorum.QuorumPeer.run( > > > > QuorumPeer.java:786) > > > > 2017-01-24 20:52:41,073 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.Learner: Getting a diff from the leader 0x1e0000258d > > > > 2017-01-24 20:52:41,116 INFO [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > persistence.FileTxnSnapLog: Snapshotting: 0x1e0000258d to > > > > /home/hdadmin/zookeeper2/version-2/snapshot.1e0000258d > > > > 2017-01-24 20:52:42,408 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.29:44336 > > > > 2017-01-24 20:52:42,418 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.29:44336 > > > > 2017-01-24 20:52:42,421 WARN [QuorumPeer[myid=3D2]/0:0:0:0:0: > > 0:0:0:2181] > > > > quorum.Learner: Got zxid 0x1f00000001 expected 0x1 > > > > 2017-01-24 20:52:42,421 INFO [SyncThread:2] persistence.FileTxnLog= : > > > > Creating new log file: log.1f00000001 > > > > 2017-01-24 20:52:42,440 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750000 with negotiated timeout 90000= for > > > > client /192.168.0.29:44336 > > > > 2017-01-24 20:52:42,734 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.22:43377 > > > > 2017-01-24 20:52:42,737 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.22:43377 > > > > 2017-01-24 20:52:42,742 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750001 with negotiated timeout 90000= for > > > > client /192.168.0.22:43377 > > > > 2017-01-24 20:52:43,257 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.23:59005 > > > > 2017-01-24 20:52:43,258 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.23:59005 > > > > 2017-01-24 20:52:43,262 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750002 with negotiated timeout 90000= for > > > > client /192.168.0.23:59005 > > > > > > > > ---------- > > > > > > > > > > > > I've manually checked connecting to the most relevant ports of HBas= e > > and > > > > ZooKeeper (using nc or telnet) and always succeeded. > > > > > > > > During a job execution, these are the type of errors that I'm getti= ng: > > > > > > > > ---------- > > > > > > > > 2017-01-24 21:10:37,187 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.29:44750 > > > > 2017-01-24 21:10:37,193 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.29:44750 > > > > 2017-01-24 21:10:37,198 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750003 with negotiated timeout 90000= for > > > > client /192.168.0.29:44750 > > > > 2017-01-24 21:10:38,136 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750003, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:10:38,137 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.29:44750 which had sessionid 0x259d20997750003 > > > > 2017-01-24 21:18:45,469 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49706 > > > > 2017-01-24 21:18:45,479 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49706 > > > > 2017-01-24 21:18:45,543 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750004 with negotiated timeout 90000= for > > > > client /192.168.0.19:49706 > > > > 2017-01-24 21:18:45,684 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49708 > > > > 2017-01-24 21:18:45,689 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49708 > > > > 2017-01-24 21:18:45,820 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49709 > > > > 2017-01-24 21:18:45,824 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49709 > > > > 2017-01-24 21:18:45,825 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750005 with negotiated timeout 90000= for > > > > client /192.168.0.19:49708 > > > > 2017-01-24 21:18:45,840 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750006 with negotiated timeout 90000= for > > > > client /192.168.0.19:49709 > > > > 2017-01-24 21:18:45,892 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49710 > > > > 2017-01-24 21:18:45,896 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49710 > > > > 2017-01-24 21:18:45,965 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750007 with negotiated timeout 90000= for > > > > client /192.168.0.19:49710 > > > > 2017-01-24 21:18:46,010 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49711 > > > > 2017-01-24 21:18:46,012 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49711 > > > > 2017-01-24 21:18:46,065 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49712 > > > > 2017-01-24 21:18:46,067 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49712 > > > > 2017-01-24 21:18:46,133 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750008 with negotiated timeout 90000= for > > > > client /192.168.0.19:49711 > > > > 2017-01-24 21:18:46,169 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750009 with negotiated timeout 90000= for > > > > client /192.168.0.19:49712 > > > > 2017-01-24 21:18:46,571 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49713 > > > > 2017-01-24 21:18:46,578 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49713 > > > > 2017-01-24 21:18:46,655 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000a with negotiated timeout 90000= for > > > > client /192.168.0.19:49713 > > > > 2017-01-24 21:18:46,875 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49714 > > > > 2017-01-24 21:18:46,877 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49714 > > > > 2017-01-24 21:18:46,894 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000b with negotiated timeout 90000= for > > > > client /192.168.0.19:49714 > > > > 2017-01-24 21:29:04,319 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750006, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,320 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49709 which had sessionid 0x259d20997750006 > > > > 2017-01-24 21:29:04,344 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000a, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,345 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49713 which had sessionid 0x259d2099775000a > > > > 2017-01-24 21:29:04,361 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750004, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,361 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49706 which had sessionid 0x259d20997750004 > > > > 2017-01-24 21:29:04,381 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750008, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,381 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49711 which had sessionid 0x259d20997750008 > > > > 2017-01-24 21:29:04,392 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000b, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,393 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49714 which had sessionid 0x259d2099775000b > > > > 2017-01-24 21:29:04,412 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750005, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,413 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49708 which had sessionid 0x259d20997750005 > > > > 2017-01-24 21:29:04,425 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750007, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,426 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49710 which had sessionid 0x259d20997750007 > > > > 2017-01-24 21:29:04,445 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750009, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:29:04,446 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49712 which had sessionid 0x259d20997750009 > > > > 2017-01-24 21:29:10,042 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49761 > > > > 2017-01-24 21:29:10,044 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49761 > > > > 2017-01-24 21:29:10,051 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000c with negotiated timeout 90000= for > > > > client /192.168.0.19:49761 > > > > 2017-01-24 21:29:12,150 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49765 > > > > 2017-01-24 21:29:12,152 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49765 > > > > 2017-01-24 21:29:12,165 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000d with negotiated timeout 90000= for > > > > client /192.168.0.19:49765 > > > > 2017-01-24 21:29:13,332 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49769 > > > > 2017-01-24 21:29:13,334 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49769 > > > > 2017-01-24 21:29:13,343 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000e with negotiated timeout 90000= for > > > > client /192.168.0.19:49769 > > > > 2017-01-24 21:29:13,933 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49772 > > > > 2017-01-24 21:29:13,935 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49772 > > > > 2017-01-24 21:29:13,939 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d2099775000f with negotiated timeout 90000= for > > > > client /192.168.0.19:49772 > > > > 2017-01-24 21:39:34,294 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000e, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:39:34,295 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49769 which had sessionid 0x259d2099775000e > > > > 2017-01-24 21:39:34,314 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000d, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:39:34,315 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49765 which had sessionid 0x259d2099775000d > > > > 2017-01-24 21:39:34,343 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000f, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:39:34,343 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49772 which had sessionid 0x259d2099775000f > > > > 2017-01-24 21:39:34,366 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d2099775000c, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:39:34,367 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49761 which had sessionid 0x259d2099775000c > > > > 2017-01-24 21:39:38,640 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49817 > > > > 2017-01-24 21:39:38,642 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49817 > > > > 2017-01-24 21:39:38,647 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750010 with negotiated timeout 90000= for > > > > client /192.168.0.19:49817 > > > > 2017-01-24 21:39:40,386 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49821 > > > > 2017-01-24 21:39:40,392 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49821 > > > > 2017-01-24 21:39:40,399 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750011 with negotiated timeout 90000= for > > > > client /192.168.0.19:49821 > > > > 2017-01-24 21:39:41,708 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49825 > > > > 2017-01-24 21:39:41,712 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49825 > > > > 2017-01-24 21:39:41,716 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750012 with negotiated timeout 90000= for > > > > client /192.168.0.19:49825 > > > > 2017-01-24 21:39:42,420 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49828 > > > > 2017-01-24 21:39:42,421 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49828 > > > > 2017-01-24 21:39:42,426 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750013 with negotiated timeout 90000= for > > > > client /192.168.0.19:49828 > > > > 2017-01-24 21:50:04,309 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750013, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:50:04,309 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49828 which had sessionid 0x259d20997750013 > > > > 2017-01-24 21:50:04,328 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750012, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:50:04,329 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49825 which had sessionid 0x259d20997750012 > > > > 2017-01-24 21:50:04,384 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750010, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:50:04,387 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49817 which had sessionid 0x259d20997750010 > > > > 2017-01-24 21:50:04,396 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750011, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 21:50:04,397 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49821 which had sessionid 0x259d20997750011 > > > > 2017-01-24 21:50:08,380 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49857 > > > > 2017-01-24 21:50:08,382 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49857 > > > > 2017-01-24 21:50:08,387 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750014 with negotiated timeout 90000= for > > > > client /192.168.0.19:49857 > > > > 2017-01-24 21:50:09,791 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49862 > > > > 2017-01-24 21:50:09,792 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49862 > > > > 2017-01-24 21:50:09,799 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750015 with negotiated timeout 90000= for > > > > client /192.168.0.19:49862 > > > > 2017-01-24 21:50:11,381 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49866 > > > > 2017-01-24 21:50:11,383 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49866 > > > > 2017-01-24 21:50:11,390 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750016 with negotiated timeout 90000= for > > > > client /192.168.0.19:49866 > > > > 2017-01-24 21:50:12,189 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxnFactory: Accepted socket connection from / > > > > 192.168.0.19:49869 > > > > 2017-01-24 21:50:12,191 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.ZooKeeperServer: Client attempting to establish new session = at / > > > > 192.168.0.19:49869 > > > > 2017-01-24 21:50:12,196 INFO [CommitProcessor:2] > > server.ZooKeeperServer: > > > > Established session 0x259d20997750017 with negotiated timeout 90000= for > > > > client /192.168.0.19:49869 > > > > 2017-01-24 22:00:34,306 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750015, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 22:00:34,307 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49862 which had sessionid 0x259d20997750015 > > > > 2017-01-24 22:00:34,330 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750017, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 22:00:34,337 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49869 which had sessionid 0x259d20997750017 > > > > 2017-01-24 22:00:34,355 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750016, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 22:00:34,356 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49866 which had sessionid 0x259d20997750016 > > > > 2017-01-24 22:00:34,369 WARN [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: caught end of stream exception > > > > EndOfStreamException: Unable to read additional data from client > > sessionid > > > > 0x259d20997750014, likely client has closed socket > > > > at org.apache.zookeeper.server.NIOServerCnxn.doIO( > > > > NIOServerCnxn.java:228) > > > > at org.apache.zookeeper.server.NIOServerCnxnFactory.run( > > > > NIOServerCnxnFactory.java:208) > > > > at java.lang.Thread.run(Thread.java:745) > > > > 2017-01-24 22:00:34,369 INFO [NIOServerCxn.Factory:0.0.0.0/ > > 0.0.0.0:2181] > > > > server.NIOServerCnxn: Closed socket connection for client / > > > > 192.168.0.19:49857 which had sessionid 0x259d20997750014 > > > > > > > > ---------- > > > > > > > > The RegionServer logs are clean, no warnings. I assume this is a > > > > ZooKeeper problem more than HBase, but I tried many different > > > > configurations and nothing worked. Does anyone have an idea what co= uld > > > > be happening? > > > > > > > > Best, > > > > Hern=C3=A1n. > > > > > > > > > > --=-qe22pRSkFwiKX8Ea3+yG--