Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BCD21200B73 for ; Mon, 29 Aug 2016 11:58:23 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id BB31B160AB8; Mon, 29 Aug 2016 09:58:23 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 68268160AA7 for ; Mon, 29 Aug 2016 11:58:21 +0200 (CEST) Received: (qmail 17140 invoked by uid 500); 29 Aug 2016 09:58:20 -0000 Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list issues@hbase.apache.org Received: (qmail 17127 invoked by uid 99); 29 Aug 2016 09:58:20 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Aug 2016 09:58:20 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 692CE2C0031 for ; Mon, 29 Aug 2016 09:58:20 +0000 (UTC) Date: Mon, 29 Aug 2016 09:58:20 +0000 (UTC) From: "Xingxing Di (JIRA)" To: issues@hbase.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HBASE-16514) RegionServer shutdown by some unknown reason. MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Mon, 29 Aug 2016 09:58:23 -0000 [ https://issues.apache.org/jira/browse/HBASE-16514?page=3Dcom.atlassi= an.jira.plugin.system.issuetabpanels:all-tabpanel ] Xingxing Di updated HBASE-16514: -------------------------------- Description:=20 The regionserver process was shutdown by some reason. I looked into regionserver's log , it looks like detected jvm pause about 9= 122ms (zk session timout is 40s), and after 10 seconds it start another pro= cess to delete regionserver's ephemeral node in zk, and regionserver proce= ss was gone, here is the detail log: {quote} 2016-08-14 22:16:53,139 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detec= ted pause in JVM or host machine (eg GC): pause of approximately 9022ms GC pool 'ConcurrentMarkSweep' had collection(s): count=3D2 time=3D9122ms 2016-08-14 22:17:03,121 WARN [main] util.HeapMemorySizeUtil: hbase.regions= erver.global.memstore.upperLimit is deprecated by hbase.regionserver.global= .memstore.size 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:zookeeper.version=3D3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:host.name=3Dolap3.data.lq 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.version=3D1.7.0_09-icedtea 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.vendor=3DOracle Corporation 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.home=3D/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.class.path=3D/data/sysdir/hbase-1.1.2/conf:/usr/lib/jvm/java-1.7.0-o= penjdk.x86_64/lib/tools.jar:/data/sysdir/hbase-1.1.2:/data/sysdir/hbase-1.1= .2/lib/activation-1.1.jar:/data/sysdir/hbase-1.1.2/lib/aopalliance-1.0.jar:= /data/sysdir/hbase-1.1.2/lib/apacheds-i18n-2.0.0-M15.jar:/data/sysdir/hbase= -1.1.2/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/data/sysdir/hbase-1.1.2/l= ib/api-asn1-api-1.0.0-M20.jar:/data/sysdir/hbase-1.1.2/lib/api-util-1.0.0-M= 20.jar:/data/sysdir/hbase-1.1.2/lib/asm-3.1.jar:/data/sysdir/hbase-1.1.2/li= b/avro-1.7.4.jar:/data/sysdir/hbase-1.1.2/lib/commons-beanutils-1.7.0.jar:/= data/sysdir/hbase-1.1.2/lib/commons-beanutils-core-1.8.0.jar:/data/sysdir/h= base-1.1.2/lib/commons-cli-1.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-cod= ec-1.9.jar:/data/sysdir/hbase-1.1.2/lib/commons-collections-3.2.1.jar:/data= /sysdir/hbase-1.1.2/lib/commons-compress-1.4.1.jar:/data/sysdir/hbase-1.1.2= /lib/commons-configuration-1.6.jar:/data/sysdir/hbase-1.1.2/lib/commons-dae= mon-1.0.13.jar:/data/sysdir/hbase-1.1.2/lib/commons-digester-1.8.jar:/data/= sysdir/hbase-1.1.2/lib/commons-el-1.0.jar:/data/sysdir/hbase-1.1.2/lib/comm= ons-httpclient-3.1.jar:/data/sysdir/hbase-1.1.2/lib/commons-io-2.4.jar:/dat= a/sysdir/hbase-1.1.2/lib/commons-lang-2.6.jar:/data/sysdir/hbase-1.1.2/lib/= commons-logging-1.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-math-2.2.jar:/= data/sysdir/hbase-1.1.2/lib/commons-math3-3.1.1.jar:/data/sysdir/hbase-1.1.= 2/lib/commons-net-3.1.jar:/data/sysdir/hbase-1.1.2/lib/disruptor-3.3.0.jar:= /data/sysdir/hbase-1.1.2/lib/findbugs-annotations-1.3.9-1.jar:/data/sysdir/= hbase-1.1.2/lib/guava-12.0.1.jar:/data/sysdir/hbase-1.1.2/lib/guice-3.0.jar= :/data/sysdir/hbase-1.1.2/lib/guice-servlet-3.0.jar:/data/sysdir/hbase-1.1.= 2/lib/hadoop-annotations-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-auth= -2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-client-2.5.1.jar:/data/sysdi= r/hbase-1.1.2/lib/hadoop-common-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hado= op-hdfs-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-app-= 2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-common-2.5.1= .jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-core-2.5.1.jar:/d= ata/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/dat= a/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/data/sy= sdir/hbase-1.1.2/lib/hadoop-yarn-api-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib= /hadoop-yarn-client-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-comm= on-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-server-common-2.5.1.j= ar:/data/sysdir/hbase-1.1.2/lib/hbase-annotations-1.1.2.jar:/data/sysdir/hb= ase-1.1.2/lib/hbase-annotations-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/li= b/hbase-client-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-common-1.1.2.ja= r:/data/sysdir/hbase-1.1.2/lib/hbase-common-1.1.2-tests.jar:/data/sysdir/hb= ase-1.1.2/lib/hbase-examples-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-h= adoop2-compat-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-hadoop-compat-1.= 1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-it-1.1.2.jar:/data/sysdir/hbase-= 1.1.2/lib/hbase-it-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib/hbase-prefi= x-tree-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-procedure-1.1.2.jar:/da= ta/sysdir/hbase-1.1.2/lib/hbase-protocol-1.1.2.jar:/data/sysdir/hbase-1.1.2= /lib/hbase-resource-bundle-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-res= t-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-server-1.1.2.jar:/data/sysdi= r/hbase-1.1.2/lib/hbase-server-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib= /hbase-shell-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-thrift-1.1.2.jar:= /data/sysdir/hbase-1.1.2/lib/htrace-core-3.1.0-incubating.jar:/data/sysdir/= hbase-1.1.2/lib/httpclient-4.2.5.jar:/data/sysdir/hbase-1.1.2/lib/httpcore-= 4.1.3.jar:/data/sysdir/hbase-1.1.2/lib/jackson-core-asl-1.9.13.jar:/data/sy= sdir/hbase-1.1.2/lib/jackson-jaxrs-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/= jackson-mapper-asl-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/jackson-xc-1.9.1= 3.jar:/data/sysdir/hbase-1.1.2/lib/jamon-runtime-2.3.1.jar:/data/sysdir/hba= se-1.1.2/lib/jasper-compiler-5.5.23.jar:/data/sysdir/hbase-1.1.2/lib/jasper= -runtime-5.5.23.jar:/data/sysdir/hbase-1.1.2/lib/javax.inject-1.jar:/data/s= ysdir/hbase-1.1.2/lib/java-xmlbuilder-0.4.jar:/data/sysdir/hbase-1.1.2/lib/= jaxb-api-2.2.2.jar:/data/sysdir/hbase-1.1.2/lib/jaxb-impl-2.2.3-1.jar:/data= /sysdir/hbase-1.1.2/lib/jcodings-1.0.8.jar:/data/sysdir/hbase-1.1.2/lib/jer= sey-client-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-core-1.9.jar:/data/s= ysdir/hbase-1.1.2/lib/jersey-guice-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jer= sey-json-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-server-1.9.jar:/data/s= ysdir/hbase-1.1.2/lib/jets3t-0.9.0.jar:/data/sysdir/hbase-1.1.2/lib/jettiso= n-1.3.3.jar:/data/sysdir/hbase-1.1.2/lib/jetty-6.1.26.jar:/data/sysdir/hbas= e-1.1.2/lib/jetty-sslengine-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/jetty-u= til-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/joni-2.1.2.jar:/data/sysdir/hba= se-1.1.2/lib/jruby-complete-1.6.8.jar:/data/sysdir/hbase-1.1.2/lib/jsch-0.1= .42.jar:/data/sysdir/hbase-1.1.2/lib/jsp-2.1-6.1.14.jar:/data/sysdir/hbase-= 1.1.2/lib/jsp-api-2.1-6.1.14.jar:/data/sysdir/hbase-1.1.2/lib/jsr305-1.3.9.= jar:/data/sysdir/hbase-1.1.2/lib/junit-4.11.jar:/data/sysdir/hbase-1.1.2/li= b/leveldbjni-all-1.8.jar:/data/sysdir/hbase-1.1.2/lib/libthrift-0.9.0.jar:/= data/sysdir/hbase-1.1.2/lib/log4j-1.2.17.jar:/data/sysdir/hbase-1.1.2/lib/m= etrics-core-2.2.0.jar:/data/sysdir/hbase-1.1.2/lib/netty-3.2.4.Final.jar:/d= ata/sysdir/hbase-1.1.2/lib/netty-all-4.0.23.Final.jar:/data/sysdir/hbase-1.= 1.2/lib/paranamer-2.3.jar:/data/sysdir/hbase-1.1.2/lib/protobuf-java-2.5.0.= jar:/data/sysdir/hbase-1.1.2/lib/servlet-api-2.5-6.1.14.jar:/data/sysdir/hb= ase-1.1.2/lib/servlet-api-2.5.jar:/data/sysdir/hbase-1.1.2/lib/slf4j-api-1.= 7.7.jar:/data/sysdir/hbase-1.1.2/lib/slf4j-log4j12-1.7.5.jar:/data/sysdir/h= base-1.1.2/lib/snappy-java-1.0.4.1.jar:/data/sysdir/hbase-1.1.2/lib/spymemc= ached-2.11.6.jar:/data/sysdir/hbase-1.1.2/lib/xmlenc-0.52.jar:/data/sysdir/= hbase-1.1.2/lib/xz-1.0.jar:/data/sysdir/hbase-1.1.2/lib/zookeeper-3.4.6.jar= :/data/sysdir/hadoop-2.4.1-hbase/etc/hadoop:/data/sysdir/hadoop-2.4.1-hbase= /share/hadoop/common/lib/commons-net-3.1.jar:/data/sysdir/hadoop-2.4.1-hbas= e/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/data/sysdir/hadoop-2.4.1-hbas= e/share/hadoop/common/lib/jets3t-0.9.0.jar:/data/sysdir/hadoop-2.4.1-hbase/= share/hadoop/common/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/= share/hadoop/common/lib/httpclient-4.2.5.jar:/data/sysdir/hadoop-2.4.1-hbas= e/share/hadoop/common/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.= 4.1-hbase/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/data/sysdir/hado= op-2.4.1-hbase/share/hadoop/common/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.= 1-hbase/share/hadoop/common/lib/commons-collections-3.2.1.jar:/data/sysdir/= hadoop-2.4.1-hbase/share/hadoop/common/lib/jettison-1.1.jar:/data/sysdir/ha= doop-2.4.1-hbase/share/hadoop/common/lib/commons-logging-1.1.3.jar:/data/sy= sdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-codec-1.4.jar:/data= /sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-digester-1.8.jar= :/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsr305-1.3.9.jar:/= data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/activation-1.1.jar:/= data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/xmlenc-0.52.jar:/dat= a/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-mapper-asl-1.8.= 8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jetty-6.1.26.= jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/log4j-1.2.17.ja= r:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/netty-3.6.2.Final= .jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/stax-api-1.0-2= .jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/hadoop-auth-2.= 4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jasper-comp= iler-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/com= mons-beanutils-core-1.8.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= common/lib/hadoop-annotations-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/sha= re/hadoop/common/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/sh= are/hadoop/common/lib/jasper-runtime-5.5.23.jar:/data/sysdir/hadoop-2.4.1-h= base/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/data/sysdir/hadoop-2.= 4.1-hbase/share/hadoop/common/lib/avro-1.7.4.jar:/data/sysdir/hadoop-2.4.1-= hbase/share/hadoop/common/lib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1= -hbase/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hado= op-2.4.1-hbase/share/hadoop/common/lib/jetty-util-6.1.26.jar:/data/sysdir/h= adoop-2.4.1-hbase/share/hadoop/common/lib/jersey-server-1.9.jar:/data/sysdi= r/hadoop-2.4.1-hbase/share/hadoop/common/lib/httpcore-4.2.5.jar:/data/sysdi= r/hadoop-2.4.1-hbase/share/hadoop/common/lib/paranamer-2.3.jar:/data/sysdir= /hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/data/s= ysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/data/= sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/= data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsch-0.1.42.jar:/dat= a/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/guava-11.0.2.jar:/data/= sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/= data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-configuratio= n-1.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/mockito-a= ll-1.8.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jersey= -json-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jacks= on-xc-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/com= mons-el-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsp= -api-2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/servle= t-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commo= ns-httpclient-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/l= ib/zookeeper-3.4.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/= lib/commons-math3-3.1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/co= mmon/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= common/lib/junit-4.8.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/com= mon/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/= jaxb-impl-2.2.3-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/l= ib/commons-beanutils-1.7.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /common/hadoop-common-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/share= /hadoop/common/hadoop-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/shar= e/hadoop/common/hadoop-common-2.4.1-path-1.jar:/data/sysdir/hadoop-2.4.1-hb= ase/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/data/sysdir/hadoop-= 2.4.1-hbase/share/hadoop/common/hadoop-nfs-2.4.1.jar:/data/sysdir/hadoop-2.= 4.1-hbase/share/hadoop/hdfs:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hd= fs/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs= /lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/comm= ons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib= /commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/li= b/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/xm= lenc-0.52.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jackson= -mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib= /jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/log= 4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/netty-3= .6.2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/common= s-daemon-1.0.13.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/j= ersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/ja= sper-runtime-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/l= ib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/li= b/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/h= dfs/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= hdfs/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /hdfs/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdf= s/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= hdfs/lib/commons-el-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hd= fs/lib/jsp-api-2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/li= b/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib= /commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/had= oop-hdfs-2.4.1-patch-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdf= s/hadoop-hdfs-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= hdfs/hadoop-hdfs-nfs-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/jaxb-api-2.2.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hado= op/yarn/lib/javax.inject-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib= /commons-collections-3.2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/jline-0.9.94.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/jettison-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/li= b/commons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/ya= rn/lib/commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/= lib/activation-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/li= b/jackson-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/log4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/li= b/leveldbjni-all-1.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/= lib/stax-api-1.0-2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/li= b/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib= /jersey-guice-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib= /commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/= jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/ya= rn/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/aopalliance-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/ya= rn/lib/jackson-jaxrs-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/guice-servlet-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoo= p/yarn/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/ya= rn/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/jersey-json-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= yarn/lib/jackson-xc-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= yarn/lib/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/commons-httpclient-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/ha= doop/yarn/lib/zookeeper-3.4.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/had= oop/yarn/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/had= oop/yarn/lib/guice-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jaxb= -impl-2.2.3-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jer= sey-client-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop= -yarn-server-applicationhistoryservice-2.4.1.jar:/data/sysdir/hadoop-2.4.1-= hbase/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.4.1.jar= :/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-nodem= anager-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-y= arn-server-resourcemanager-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/= hadoop/yarn/hadoop-yarn-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/sh= are/hadoop/yarn/hadoop-yarn-server-tests-2.4.1.jar:/data/sysdir/hadoop-2.4.= 1-hbase/share/hadoop/yarn/hadoop-yarn-client-2.4.1.jar:/data/sysdir/hadoop-= 2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.4.1.jar:/data/= sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-applications-unmana= ged-am-launcher-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn= /hadoop-yarn-server-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/= hadoop/yarn/hadoop-yarn-api-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share= /hadoop/mapreduce/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/sh= are/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.4= .1-hbase/share/hadoop/mapreduce/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar:/data/= sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/javax.inject-1.jar:/da= ta/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/asm-3.2.jar:/data/s= ysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.= 8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/log4j-1.2.= 17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/netty-3.6= .2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/had= oop-annotations-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapr= educe/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= mapreduce/lib/jersey-guice-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/ha= doop/mapreduce/lib/avro-1.7.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/had= oop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbas= e/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.= 1-hbase/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/data/sysdir/hadoop-2.= 4.1-hbase/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/data/sysdir/hadoo= p-2.4.1-hbase/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/data/sysdir= /hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/dat= a/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/hamcrest-core-1.1.ja= r:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/protobuf-java-= 2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/junit-= 4.10.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/guice-3= .0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/xz-1.0.ja= r:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-e= xamples-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/ha= doop-mapreduce-client-app-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/h= adoop/mapreduce/hadoop-mapreduce-client-common-2.4.1.jar:/data/sysdir/hadoo= p-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.= 1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapredu= ce-client-core-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapre= duce/hadoop-mapreduce-client-shuffle-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hb= ase/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.4.1.jar:/data/sysdi= r/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plug= ins-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop= -mapreduce-client-jobclient-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase= /contrib/capacity-scheduler/*.jar 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.library.path=3D/data/sysdir/hadoop-2.4.1-hbase/lib/native 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.io.tmpdir=3D/tmp 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.compiler=3D 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:os.name=3DLinux 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:os.arch=3Damd64 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:os.version=3D3.13.6 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:user.name=3Dhbase 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:user.home=3D/data/home/hbase 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:user.dir=3D/data/sysdir/hbase-1.1.2/conf 2016-08-14 22:17:03,180 INFO [main] zookeeper.ZooKeeper: Initiating client= connection, connectString=3D192.168.201.52:2181,192.168.201.53:2181,192.16= 8.201.63:2181,192.168.201.64:2181,192.168.201.65:2181 sessionTimeout=3D3000= 0 watcher=3Dorg.apache.zookeeper.ZooKeeperMain$MyWatcher@38987d26 2016-08-14 22:17:03,206 INFO [main-SendThread(192.168.201.65:2181)] zookee= per.ClientCnxn: Opening socket connection to server 192.168.201.65/192.168.= 201.65:2181. Will not attempt to authenticate using SASL (unknown error) 2016-08-14 22:17:03,228 INFO [main-SendThread(192.168.201.65:2181)] zookee= per.ClientCnxn: Socket connection established to 192.168.201.65/192.168.201= .65:2181, initiating session 2016-08-14 22:17:03,234 INFO [main-SendThread(192.168.201.65:2181)] zookee= per.ClientCnxn: Session establishment complete on server 192.168.201.65/192= .168.201.65:2181, sessionid =3D 0x556683846220a79, negotiated timeout =3D 3= 0000 {quote} Then i looked into zk's log, i found these : {quote} 8/14/16 10:17:03 PM CST session 0x556683846220a79 cxid 0x0 zxid 0xf73ca5859= createSession 30000 8/14/16 10:17:03 PM CST session 0x556683846220a79 cxid 0x1 zxid 0xf73ca585a= delete '/olapHbase/rs/olap3.data.lq%2C16020%2C1470799848293 {quote} And i looked master logs as below :=20 {quote} 2016-08-14 22:16:43,942 INFO [ProcedureExecutorThread-0] zookeeper.ZKTable= StateManager: Moving table KYLIN_F5O605Z3QZ state from null to ENABLING 2016-08-14 22:16:43,952 INFO [ProcedureExecutorThread-0] master.Assignment= Manager: Bulk assigning 1 region(s) across 4 server(s), round-robin=3Dtrue 2016-08-14 22:16:43,953 INFO [olap1.data.lq,16000,1469605579716-GeneralBul= kAssigner-2] master.AssignmentManager: Assigning 1 region(s) to olap3.data.= lq,16020,1470799848293 2016-08-14 22:16:43,958 INFO [olap1.data.lq,16000,1469605579716-GeneralBul= kAssigner-2] master.RegionStates: Transition {6e207f0307d4213d6c0e195115df1= b8d state=3DOFFLINE, ts=3D1471184203953, server=3Dnull} to {6e207f0307d4213= d6c0e195115df1b8d state=3DPENDING_OPEN, ts=3D1471184203958, server=3Dolap3.= data.lq,16020,1470799848293} 2016-08-14 22:16:43,985 INFO [AM.ZK.Worker-pool2-t4001] master.RegionState= s: Transition {6e207f0307d4213d6c0e195115df1b8d state=3DPENDING_OPEN, ts=3D= 1471184203958, server=3Dolap3.data.lq,16020,1470799848293} to {6e207f0307d4= 213d6c0e195115df1b8d state=3DOPENING, ts=3D1471184203985, server=3Dolap3.da= ta.lq,16020,1470799848293} 2016-08-14 22:17:03,238 INFO [main-EventThread] zookeeper.RegionServerTrac= ker: RegionServer ephemeral node deleted, processing expiration [olap3.data= .lq,16020,1470799848293] 2016-08-14 22:17:03,240 WARN [MASTER_SERVER_OPERATIONS-olap1:16000-2] hbas= e.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is de= precated. Instead, use "hbase.client.scanner.timeout.period" 2016-08-14 22:17:04,008 INFO [ProcedureExecutorThread-0] master.Assignment= Manager: Bulk assigning done 2016-08-14 22:17:04,011 INFO [ProcedureExecutorThread-0] zookeeper.ZKTable= StateManager: Moving table KYLIN_F5O605Z3QZ state from ENABLING to ENABLED 2016-08-14 22:17:04,251 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] hand= ler.ServerShutdownHandler: Splitting logs for olap3.data.lq,16020,147079984= 8293 before assignment; region count=3D2033 2016-08-14 22:17:04,261 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] mast= er.SplitLogManager: dead splitlog workers [olap3.data.lq,16020,147079984829= 3] 2016-08-14 22:17:04,263 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] mast= er.SplitLogManager: started splitting 2 logs in [hdfs://olapHbaseCluster:80= 20/hbase/WALs/olap3.data.lq,16020,1470799848293-splitting] for [olap3.data.= lq,16020,1470799848293] 2016-08-14 22:17:04,268 INFO [main-EventThread] coordination.SplitLogManag= erCoordination: task /olapHbase/splitWAL/WALs%2Folap3.data.lq%2C16020%2C147= 0799848293-splitting%2Folap3.data.lq%252C16020%252C1470799848293.default.14= 71182516751 acquired by olap1.data.lq,16020,1470799983323 2016-08-14 22:17:04,277 INFO [main-EventThread] coordination.SplitLogManag= erCoordination: task /olapHbase/splitWAL/WALs%2Folap3.data.lq%2C16020%2C147= 0799848293-splitting%2Folap3.data.lq%252C16020%252C1470799848293.default.14= 71180125256 acquired by olap2.data.lq,16020,1470800075996 2016-08-14 22:17:04,288 WARN [olap1.data.lq,16000,1469605579716_ChoreServi= ce_1] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.per= iod" is deprecated. Instead, use "hbase.client.scanner.timeout.period" .... {quote} I'm not falimliar with hbase , may be it's not a bug , but i can not figure= out why this happend. I will be grateful if somebody can help me. was: The regionserver process was shutdown by some reason. I looked into regionserver's log , it looks like detected jvm pause, and af= ter 10 seconds it start another process to delete regionserver's ephemeral= node in zk, and regionserver process was gone, here is the detail log: {quote} 2016-08-14 22:16:53,139 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detec= ted pause in JVM or host machine (eg GC): pause of approximately 9022ms GC pool 'ConcurrentMarkSweep' had collection(s): count=3D2 time=3D9122ms 2016-08-14 22:17:03,121 WARN [main] util.HeapMemorySizeUtil: hbase.regions= erver.global.memstore.upperLimit is deprecated by hbase.regionserver.global= .memstore.size 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:zookeeper.version=3D3.4.6-1569965, built on 02/20/2014 09:09 GMT 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:host.name=3Dolap3.data.lq 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.version=3D1.7.0_09-icedtea 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.vendor=3DOracle Corporation 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.home=3D/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.class.path=3D/data/sysdir/hbase-1.1.2/conf:/usr/lib/jvm/java-1.7.0-o= penjdk.x86_64/lib/tools.jar:/data/sysdir/hbase-1.1.2:/data/sysdir/hbase-1.1= .2/lib/activation-1.1.jar:/data/sysdir/hbase-1.1.2/lib/aopalliance-1.0.jar:= /data/sysdir/hbase-1.1.2/lib/apacheds-i18n-2.0.0-M15.jar:/data/sysdir/hbase= -1.1.2/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/data/sysdir/hbase-1.1.2/l= ib/api-asn1-api-1.0.0-M20.jar:/data/sysdir/hbase-1.1.2/lib/api-util-1.0.0-M= 20.jar:/data/sysdir/hbase-1.1.2/lib/asm-3.1.jar:/data/sysdir/hbase-1.1.2/li= b/avro-1.7.4.jar:/data/sysdir/hbase-1.1.2/lib/commons-beanutils-1.7.0.jar:/= data/sysdir/hbase-1.1.2/lib/commons-beanutils-core-1.8.0.jar:/data/sysdir/h= base-1.1.2/lib/commons-cli-1.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-cod= ec-1.9.jar:/data/sysdir/hbase-1.1.2/lib/commons-collections-3.2.1.jar:/data= /sysdir/hbase-1.1.2/lib/commons-compress-1.4.1.jar:/data/sysdir/hbase-1.1.2= /lib/commons-configuration-1.6.jar:/data/sysdir/hbase-1.1.2/lib/commons-dae= mon-1.0.13.jar:/data/sysdir/hbase-1.1.2/lib/commons-digester-1.8.jar:/data/= sysdir/hbase-1.1.2/lib/commons-el-1.0.jar:/data/sysdir/hbase-1.1.2/lib/comm= ons-httpclient-3.1.jar:/data/sysdir/hbase-1.1.2/lib/commons-io-2.4.jar:/dat= a/sysdir/hbase-1.1.2/lib/commons-lang-2.6.jar:/data/sysdir/hbase-1.1.2/lib/= commons-logging-1.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-math-2.2.jar:/= data/sysdir/hbase-1.1.2/lib/commons-math3-3.1.1.jar:/data/sysdir/hbase-1.1.= 2/lib/commons-net-3.1.jar:/data/sysdir/hbase-1.1.2/lib/disruptor-3.3.0.jar:= /data/sysdir/hbase-1.1.2/lib/findbugs-annotations-1.3.9-1.jar:/data/sysdir/= hbase-1.1.2/lib/guava-12.0.1.jar:/data/sysdir/hbase-1.1.2/lib/guice-3.0.jar= :/data/sysdir/hbase-1.1.2/lib/guice-servlet-3.0.jar:/data/sysdir/hbase-1.1.= 2/lib/hadoop-annotations-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-auth= -2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-client-2.5.1.jar:/data/sysdi= r/hbase-1.1.2/lib/hadoop-common-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hado= op-hdfs-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-app-= 2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-common-2.5.1= .jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-core-2.5.1.jar:/d= ata/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/dat= a/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/data/sy= sdir/hbase-1.1.2/lib/hadoop-yarn-api-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib= /hadoop-yarn-client-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-comm= on-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-server-common-2.5.1.j= ar:/data/sysdir/hbase-1.1.2/lib/hbase-annotations-1.1.2.jar:/data/sysdir/hb= ase-1.1.2/lib/hbase-annotations-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/li= b/hbase-client-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-common-1.1.2.ja= r:/data/sysdir/hbase-1.1.2/lib/hbase-common-1.1.2-tests.jar:/data/sysdir/hb= ase-1.1.2/lib/hbase-examples-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-h= adoop2-compat-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-hadoop-compat-1.= 1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-it-1.1.2.jar:/data/sysdir/hbase-= 1.1.2/lib/hbase-it-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib/hbase-prefi= x-tree-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-procedure-1.1.2.jar:/da= ta/sysdir/hbase-1.1.2/lib/hbase-protocol-1.1.2.jar:/data/sysdir/hbase-1.1.2= /lib/hbase-resource-bundle-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-res= t-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-server-1.1.2.jar:/data/sysdi= r/hbase-1.1.2/lib/hbase-server-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib= /hbase-shell-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-thrift-1.1.2.jar:= /data/sysdir/hbase-1.1.2/lib/htrace-core-3.1.0-incubating.jar:/data/sysdir/= hbase-1.1.2/lib/httpclient-4.2.5.jar:/data/sysdir/hbase-1.1.2/lib/httpcore-= 4.1.3.jar:/data/sysdir/hbase-1.1.2/lib/jackson-core-asl-1.9.13.jar:/data/sy= sdir/hbase-1.1.2/lib/jackson-jaxrs-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/= jackson-mapper-asl-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/jackson-xc-1.9.1= 3.jar:/data/sysdir/hbase-1.1.2/lib/jamon-runtime-2.3.1.jar:/data/sysdir/hba= se-1.1.2/lib/jasper-compiler-5.5.23.jar:/data/sysdir/hbase-1.1.2/lib/jasper= -runtime-5.5.23.jar:/data/sysdir/hbase-1.1.2/lib/javax.inject-1.jar:/data/s= ysdir/hbase-1.1.2/lib/java-xmlbuilder-0.4.jar:/data/sysdir/hbase-1.1.2/lib/= jaxb-api-2.2.2.jar:/data/sysdir/hbase-1.1.2/lib/jaxb-impl-2.2.3-1.jar:/data= /sysdir/hbase-1.1.2/lib/jcodings-1.0.8.jar:/data/sysdir/hbase-1.1.2/lib/jer= sey-client-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-core-1.9.jar:/data/s= ysdir/hbase-1.1.2/lib/jersey-guice-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jer= sey-json-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-server-1.9.jar:/data/s= ysdir/hbase-1.1.2/lib/jets3t-0.9.0.jar:/data/sysdir/hbase-1.1.2/lib/jettiso= n-1.3.3.jar:/data/sysdir/hbase-1.1.2/lib/jetty-6.1.26.jar:/data/sysdir/hbas= e-1.1.2/lib/jetty-sslengine-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/jetty-u= til-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/joni-2.1.2.jar:/data/sysdir/hba= se-1.1.2/lib/jruby-complete-1.6.8.jar:/data/sysdir/hbase-1.1.2/lib/jsch-0.1= .42.jar:/data/sysdir/hbase-1.1.2/lib/jsp-2.1-6.1.14.jar:/data/sysdir/hbase-= 1.1.2/lib/jsp-api-2.1-6.1.14.jar:/data/sysdir/hbase-1.1.2/lib/jsr305-1.3.9.= jar:/data/sysdir/hbase-1.1.2/lib/junit-4.11.jar:/data/sysdir/hbase-1.1.2/li= b/leveldbjni-all-1.8.jar:/data/sysdir/hbase-1.1.2/lib/libthrift-0.9.0.jar:/= data/sysdir/hbase-1.1.2/lib/log4j-1.2.17.jar:/data/sysdir/hbase-1.1.2/lib/m= etrics-core-2.2.0.jar:/data/sysdir/hbase-1.1.2/lib/netty-3.2.4.Final.jar:/d= ata/sysdir/hbase-1.1.2/lib/netty-all-4.0.23.Final.jar:/data/sysdir/hbase-1.= 1.2/lib/paranamer-2.3.jar:/data/sysdir/hbase-1.1.2/lib/protobuf-java-2.5.0.= jar:/data/sysdir/hbase-1.1.2/lib/servlet-api-2.5-6.1.14.jar:/data/sysdir/hb= ase-1.1.2/lib/servlet-api-2.5.jar:/data/sysdir/hbase-1.1.2/lib/slf4j-api-1.= 7.7.jar:/data/sysdir/hbase-1.1.2/lib/slf4j-log4j12-1.7.5.jar:/data/sysdir/h= base-1.1.2/lib/snappy-java-1.0.4.1.jar:/data/sysdir/hbase-1.1.2/lib/spymemc= ached-2.11.6.jar:/data/sysdir/hbase-1.1.2/lib/xmlenc-0.52.jar:/data/sysdir/= hbase-1.1.2/lib/xz-1.0.jar:/data/sysdir/hbase-1.1.2/lib/zookeeper-3.4.6.jar= :/data/sysdir/hadoop-2.4.1-hbase/etc/hadoop:/data/sysdir/hadoop-2.4.1-hbase= /share/hadoop/common/lib/commons-net-3.1.jar:/data/sysdir/hadoop-2.4.1-hbas= e/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/data/sysdir/hadoop-2.4.1-hbas= e/share/hadoop/common/lib/jets3t-0.9.0.jar:/data/sysdir/hadoop-2.4.1-hbase/= share/hadoop/common/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/= share/hadoop/common/lib/httpclient-4.2.5.jar:/data/sysdir/hadoop-2.4.1-hbas= e/share/hadoop/common/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.= 4.1-hbase/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/data/sysdir/hado= op-2.4.1-hbase/share/hadoop/common/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.= 1-hbase/share/hadoop/common/lib/commons-collections-3.2.1.jar:/data/sysdir/= hadoop-2.4.1-hbase/share/hadoop/common/lib/jettison-1.1.jar:/data/sysdir/ha= doop-2.4.1-hbase/share/hadoop/common/lib/commons-logging-1.1.3.jar:/data/sy= sdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-codec-1.4.jar:/data= /sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-digester-1.8.jar= :/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsr305-1.3.9.jar:/= data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/activation-1.1.jar:/= data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/xmlenc-0.52.jar:/dat= a/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-mapper-asl-1.8.= 8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jetty-6.1.26.= jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/log4j-1.2.17.ja= r:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/netty-3.6.2.Final= .jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/stax-api-1.0-2= .jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/hadoop-auth-2.= 4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jasper-comp= iler-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/com= mons-beanutils-core-1.8.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= common/lib/hadoop-annotations-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/sha= re/hadoop/common/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/sh= are/hadoop/common/lib/jasper-runtime-5.5.23.jar:/data/sysdir/hadoop-2.4.1-h= base/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/data/sysdir/hadoop-2.= 4.1-hbase/share/hadoop/common/lib/avro-1.7.4.jar:/data/sysdir/hadoop-2.4.1-= hbase/share/hadoop/common/lib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1= -hbase/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hado= op-2.4.1-hbase/share/hadoop/common/lib/jetty-util-6.1.26.jar:/data/sysdir/h= adoop-2.4.1-hbase/share/hadoop/common/lib/jersey-server-1.9.jar:/data/sysdi= r/hadoop-2.4.1-hbase/share/hadoop/common/lib/httpcore-4.2.5.jar:/data/sysdi= r/hadoop-2.4.1-hbase/share/hadoop/common/lib/paranamer-2.3.jar:/data/sysdir= /hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/data/s= ysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/data/= sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/snappy-java-1.0.4.1.jar:/= data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsch-0.1.42.jar:/dat= a/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/guava-11.0.2.jar:/data/= sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/protobuf-java-2.5.0.jar:/= data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-configuratio= n-1.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/mockito-a= ll-1.8.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jersey= -json-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jacks= on-xc-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/com= mons-el-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsp= -api-2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/servle= t-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commo= ns-httpclient-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/l= ib/zookeeper-3.4.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/= lib/commons-math3-3.1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/co= mmon/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= common/lib/junit-4.8.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/com= mon/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/= jaxb-impl-2.2.3-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/l= ib/commons-beanutils-1.7.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /common/hadoop-common-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/share= /hadoop/common/hadoop-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/shar= e/hadoop/common/hadoop-common-2.4.1-path-1.jar:/data/sysdir/hadoop-2.4.1-hb= ase/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/data/sysdir/hadoop-= 2.4.1-hbase/share/hadoop/common/hadoop-nfs-2.4.1.jar:/data/sysdir/hadoop-2.= 4.1-hbase/share/hadoop/hdfs:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hd= fs/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs= /lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/comm= ons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib= /commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/li= b/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/xm= lenc-0.52.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jackson= -mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib= /jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/log= 4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/netty-3= .6.2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/common= s-daemon-1.0.13.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/j= ersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/ja= sper-runtime-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/l= ib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/li= b/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/h= dfs/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= hdfs/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /hdfs/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdf= s/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= hdfs/lib/commons-el-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hd= fs/lib/jsp-api-2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/li= b/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib= /commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/had= oop-hdfs-2.4.1-patch-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdf= s/hadoop-hdfs-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= hdfs/hadoop-hdfs-nfs-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/jaxb-api-2.2.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hado= op/yarn/lib/javax.inject-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib= /commons-collections-3.2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/jline-0.9.94.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/jettison-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/li= b/commons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/ya= rn/lib/commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/= lib/activation-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/li= b/jackson-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/log4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/li= b/leveldbjni-all-1.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/= lib/stax-api-1.0-2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/li= b/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib= /jersey-guice-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib= /commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/= jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/ya= rn/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/aopalliance-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/ya= rn/lib/jackson-jaxrs-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/guice-servlet-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoo= p/yarn/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/ya= rn/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/jersey-json-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= yarn/lib/jackson-xc-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= yarn/lib/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/commons-httpclient-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/ha= doop/yarn/lib/zookeeper-3.4.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/had= oop/yarn/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/had= oop/yarn/lib/guice-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jaxb= -impl-2.2.3-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/jer= sey-client-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop= -yarn-server-applicationhistoryservice-2.4.1.jar:/data/sysdir/hadoop-2.4.1-= hbase/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.4.1.jar= :/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-nodem= anager-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-y= arn-server-resourcemanager-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/= hadoop/yarn/hadoop-yarn-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/sh= are/hadoop/yarn/hadoop-yarn-server-tests-2.4.1.jar:/data/sysdir/hadoop-2.4.= 1-hbase/share/hadoop/yarn/hadoop-yarn-client-2.4.1.jar:/data/sysdir/hadoop-= 2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.4.1.jar:/data/= sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-applications-unmana= ged-am-launcher-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn= /hadoop-yarn-server-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/= hadoop/yarn/hadoop-yarn-api-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share= /hadoop/mapreduce/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/sh= are/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.4= .1-hbase/share/hadoop/mapreduce/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar:/data/= sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/javax.inject-1.jar:/da= ta/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/asm-3.2.jar:/data/s= ysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.8.= 8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/log4j-1.2.= 17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/netty-3.6= .2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/had= oop-annotations-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapr= educe/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= mapreduce/lib/jersey-guice-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/ha= doop/mapreduce/lib/avro-1.7.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/had= oop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbas= e/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.= 1-hbase/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/data/sysdir/hadoop-2.= 4.1-hbase/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/data/sysdir/hadoo= p-2.4.1-hbase/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/data/sysdir= /hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/dat= a/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/hamcrest-core-1.1.ja= r:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/protobuf-java-= 2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/junit-= 4.10.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/guice-3= .0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/xz-1.0.ja= r:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-e= xamples-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/ha= doop-mapreduce-client-app-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/h= adoop/mapreduce/hadoop-mapreduce-client-common-2.4.1.jar:/data/sysdir/hadoo= p-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.4.= 1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapredu= ce-client-core-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapre= duce/hadoop-mapreduce-client-shuffle-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hb= ase/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.4.1.jar:/data/sysdi= r/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plug= ins-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop= -mapreduce-client-jobclient-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase= /contrib/capacity-scheduler/*.jar 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.library.path=3D/data/sysdir/hadoop-2.4.1-hbase/lib/native 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.io.tmpdir=3D/tmp 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:java.compiler=3D 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:os.name=3DLinux 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:os.arch=3Damd64 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:os.version=3D3.13.6 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:user.name=3Dhbase 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:user.home=3D/data/home/hbase 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environmen= t:user.dir=3D/data/sysdir/hbase-1.1.2/conf 2016-08-14 22:17:03,180 INFO [main] zookeeper.ZooKeeper: Initiating client= connection, connectString=3D192.168.201.52:2181,192.168.201.53:2181,192.16= 8.201.63:2181,192.168.201.64:2181,192.168.201.65:2181 sessionTimeout=3D3000= 0 watcher=3Dorg.apache.zookeeper.ZooKeeperMain$MyWatcher@38987d26 2016-08-14 22:17:03,206 INFO [main-SendThread(192.168.201.65:2181)] zookee= per.ClientCnxn: Opening socket connection to server 192.168.201.65/192.168.= 201.65:2181. Will not attempt to authenticate using SASL (unknown error) 2016-08-14 22:17:03,228 INFO [main-SendThread(192.168.201.65:2181)] zookee= per.ClientCnxn: Socket connection established to 192.168.201.65/192.168.201= .65:2181, initiating session 2016-08-14 22:17:03,234 INFO [main-SendThread(192.168.201.65:2181)] zookee= per.ClientCnxn: Session establishment complete on server 192.168.201.65/192= .168.201.65:2181, sessionid =3D 0x556683846220a79, negotiated timeout =3D 3= 0000 {quote} Then i looked into zk's log, i found these : {quote} 8/14/16 10:17:03 PM CST session 0x556683846220a79 cxid 0x0 zxid 0xf73ca5859= createSession 30000 8/14/16 10:17:03 PM CST session 0x556683846220a79 cxid 0x1 zxid 0xf73ca585a= delete '/olapHbase/rs/olap3.data.lq%2C16020%2C1470799848293 {quote} And i looked master logs as below :=20 {quote} 2016-08-14 22:16:43,942 INFO [ProcedureExecutorThread-0] zookeeper.ZKTable= StateManager: Moving table KYLIN_F5O605Z3QZ state from null to ENABLING 2016-08-14 22:16:43,952 INFO [ProcedureExecutorThread-0] master.Assignment= Manager: Bulk assigning 1 region(s) across 4 server(s), round-robin=3Dtrue 2016-08-14 22:16:43,953 INFO [olap1.data.lq,16000,1469605579716-GeneralBul= kAssigner-2] master.AssignmentManager: Assigning 1 region(s) to olap3.data.= lq,16020,1470799848293 2016-08-14 22:16:43,958 INFO [olap1.data.lq,16000,1469605579716-GeneralBul= kAssigner-2] master.RegionStates: Transition {6e207f0307d4213d6c0e195115df1= b8d state=3DOFFLINE, ts=3D1471184203953, server=3Dnull} to {6e207f0307d4213= d6c0e195115df1b8d state=3DPENDING_OPEN, ts=3D1471184203958, server=3Dolap3.= data.lq,16020,1470799848293} 2016-08-14 22:16:43,985 INFO [AM.ZK.Worker-pool2-t4001] master.RegionState= s: Transition {6e207f0307d4213d6c0e195115df1b8d state=3DPENDING_OPEN, ts=3D= 1471184203958, server=3Dolap3.data.lq,16020,1470799848293} to {6e207f0307d4= 213d6c0e195115df1b8d state=3DOPENING, ts=3D1471184203985, server=3Dolap3.da= ta.lq,16020,1470799848293} 2016-08-14 22:17:03,238 INFO [main-EventThread] zookeeper.RegionServerTrac= ker: RegionServer ephemeral node deleted, processing expiration [olap3.data= .lq,16020,1470799848293] 2016-08-14 22:17:03,240 WARN [MASTER_SERVER_OPERATIONS-olap1:16000-2] hbas= e.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is de= precated. Instead, use "hbase.client.scanner.timeout.period" 2016-08-14 22:17:04,008 INFO [ProcedureExecutorThread-0] master.Assignment= Manager: Bulk assigning done 2016-08-14 22:17:04,011 INFO [ProcedureExecutorThread-0] zookeeper.ZKTable= StateManager: Moving table KYLIN_F5O605Z3QZ state from ENABLING to ENABLED 2016-08-14 22:17:04,251 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] hand= ler.ServerShutdownHandler: Splitting logs for olap3.data.lq,16020,147079984= 8293 before assignment; region count=3D2033 2016-08-14 22:17:04,261 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] mast= er.SplitLogManager: dead splitlog workers [olap3.data.lq,16020,147079984829= 3] 2016-08-14 22:17:04,263 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] mast= er.SplitLogManager: started splitting 2 logs in [hdfs://olapHbaseCluster:80= 20/hbase/WALs/olap3.data.lq,16020,1470799848293-splitting] for [olap3.data.= lq,16020,1470799848293] 2016-08-14 22:17:04,268 INFO [main-EventThread] coordination.SplitLogManag= erCoordination: task /olapHbase/splitWAL/WALs%2Folap3.data.lq%2C16020%2C147= 0799848293-splitting%2Folap3.data.lq%252C16020%252C1470799848293.default.14= 71182516751 acquired by olap1.data.lq,16020,1470799983323 2016-08-14 22:17:04,277 INFO [main-EventThread] coordination.SplitLogManag= erCoordination: task /olapHbase/splitWAL/WALs%2Folap3.data.lq%2C16020%2C147= 0799848293-splitting%2Folap3.data.lq%252C16020%252C1470799848293.default.14= 71180125256 acquired by olap2.data.lq,16020,1470800075996 2016-08-14 22:17:04,288 WARN [olap1.data.lq,16000,1469605579716_ChoreServi= ce_1] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.per= iod" is deprecated. Instead, use "hbase.client.scanner.timeout.period" .... {quote} I'm not falimliar with hbase , may be it's not a bug , but i can not figure= out why this happend. I will be grateful if somebody can help me. > RegionServer shutdown by some unknown reason. > --------------------------------------------- > > Key: HBASE-16514 > URL: https://issues.apache.org/jira/browse/HBASE-16514 > Project: HBase > Issue Type: Bug > Components: regionserver > Affects Versions: 1.1.2 > Environment: apache hadoop 2.4.1, hbase 1.1.2 > Reporter: Xingxing Di > > The regionserver process was shutdown by some reason. > I looked into regionserver's log , it looks like detected jvm pause about= 9122ms (zk session timout is 40s), and after 10 seconds it start another p= rocess to delete regionserver's ephemeral node in zk, and regionserver pro= cess was gone, here is the detail log: > {quote} > 2016-08-14 22:16:53,139 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Det= ected pause in JVM or host machine (eg GC): pause of approximately 9022ms > GC pool 'ConcurrentMarkSweep' had collection(s): count=3D2 time=3D9122ms > 2016-08-14 22:17:03,121 WARN [main] util.HeapMemorySizeUtil: hbase.regio= nserver.global.memstore.upperLimit is deprecated by hbase.regionserver.glob= al.memstore.size > 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environm= ent:zookeeper.version=3D3.4.6-1569965, built on 02/20/2014 09:09 GMT > 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environm= ent:host.name=3Dolap3.data.lq > 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environm= ent:java.version=3D1.7.0_09-icedtea > 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environm= ent:java.vendor=3DOracle Corporation > 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environm= ent:java.home=3D/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64/jre > 2016-08-14 22:17:03,177 INFO [main] zookeeper.ZooKeeper: Client environm= ent:java.class.path=3D/data/sysdir/hbase-1.1.2/conf:/usr/lib/jvm/java-1.7.0= -openjdk.x86_64/lib/tools.jar:/data/sysdir/hbase-1.1.2:/data/sysdir/hbase-1= .1.2/lib/activation-1.1.jar:/data/sysdir/hbase-1.1.2/lib/aopalliance-1.0.ja= r:/data/sysdir/hbase-1.1.2/lib/apacheds-i18n-2.0.0-M15.jar:/data/sysdir/hba= se-1.1.2/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/data/sysdir/hbase-1.1.2= /lib/api-asn1-api-1.0.0-M20.jar:/data/sysdir/hbase-1.1.2/lib/api-util-1.0.0= -M20.jar:/data/sysdir/hbase-1.1.2/lib/asm-3.1.jar:/data/sysdir/hbase-1.1.2/= lib/avro-1.7.4.jar:/data/sysdir/hbase-1.1.2/lib/commons-beanutils-1.7.0.jar= :/data/sysdir/hbase-1.1.2/lib/commons-beanutils-core-1.8.0.jar:/data/sysdir= /hbase-1.1.2/lib/commons-cli-1.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-c= odec-1.9.jar:/data/sysdir/hbase-1.1.2/lib/commons-collections-3.2.1.jar:/da= ta/sysdir/hbase-1.1.2/lib/commons-compress-1.4.1.jar:/data/sysdir/hbase-1.1= .2/lib/commons-configuration-1.6.jar:/data/sysdir/hbase-1.1.2/lib/commons-d= aemon-1.0.13.jar:/data/sysdir/hbase-1.1.2/lib/commons-digester-1.8.jar:/dat= a/sysdir/hbase-1.1.2/lib/commons-el-1.0.jar:/data/sysdir/hbase-1.1.2/lib/co= mmons-httpclient-3.1.jar:/data/sysdir/hbase-1.1.2/lib/commons-io-2.4.jar:/d= ata/sysdir/hbase-1.1.2/lib/commons-lang-2.6.jar:/data/sysdir/hbase-1.1.2/li= b/commons-logging-1.2.jar:/data/sysdir/hbase-1.1.2/lib/commons-math-2.2.jar= :/data/sysdir/hbase-1.1.2/lib/commons-math3-3.1.1.jar:/data/sysdir/hbase-1.= 1.2/lib/commons-net-3.1.jar:/data/sysdir/hbase-1.1.2/lib/disruptor-3.3.0.ja= r:/data/sysdir/hbase-1.1.2/lib/findbugs-annotations-1.3.9-1.jar:/data/sysdi= r/hbase-1.1.2/lib/guava-12.0.1.jar:/data/sysdir/hbase-1.1.2/lib/guice-3.0.j= ar:/data/sysdir/hbase-1.1.2/lib/guice-servlet-3.0.jar:/data/sysdir/hbase-1.= 1.2/lib/hadoop-annotations-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-au= th-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-client-2.5.1.jar:/data/sys= dir/hbase-1.1.2/lib/hadoop-common-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/ha= doop-hdfs-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-ap= p-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-common-2.5= .1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-core-2.5.1.jar:= /data/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-jobclient-2.5.1.jar:/d= ata/sysdir/hbase-1.1.2/lib/hadoop-mapreduce-client-shuffle-2.5.1.jar:/data/= sysdir/hbase-1.1.2/lib/hadoop-yarn-api-2.5.1.jar:/data/sysdir/hbase-1.1.2/l= ib/hadoop-yarn-client-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-co= mmon-2.5.1.jar:/data/sysdir/hbase-1.1.2/lib/hadoop-yarn-server-common-2.5.1= .jar:/data/sysdir/hbase-1.1.2/lib/hbase-annotations-1.1.2.jar:/data/sysdir/= hbase-1.1.2/lib/hbase-annotations-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/= lib/hbase-client-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-common-1.1.2.= jar:/data/sysdir/hbase-1.1.2/lib/hbase-common-1.1.2-tests.jar:/data/sysdir/= hbase-1.1.2/lib/hbase-examples-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase= -hadoop2-compat-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-hadoop-compat-= 1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-it-1.1.2.jar:/data/sysdir/hbas= e-1.1.2/lib/hbase-it-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/lib/hbase-pre= fix-tree-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-procedure-1.1.2.jar:/= data/sysdir/hbase-1.1.2/lib/hbase-protocol-1.1.2.jar:/data/sysdir/hbase-1.1= .2/lib/hbase-resource-bundle-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-r= est-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-server-1.1.2.jar:/data/sys= dir/hbase-1.1.2/lib/hbase-server-1.1.2-tests.jar:/data/sysdir/hbase-1.1.2/l= ib/hbase-shell-1.1.2.jar:/data/sysdir/hbase-1.1.2/lib/hbase-thrift-1.1.2.ja= r:/data/sysdir/hbase-1.1.2/lib/htrace-core-3.1.0-incubating.jar:/data/sysdi= r/hbase-1.1.2/lib/httpclient-4.2.5.jar:/data/sysdir/hbase-1.1.2/lib/httpcor= e-4.1.3.jar:/data/sysdir/hbase-1.1.2/lib/jackson-core-asl-1.9.13.jar:/data/= sysdir/hbase-1.1.2/lib/jackson-jaxrs-1.9.13.jar:/data/sysdir/hbase-1.1.2/li= b/jackson-mapper-asl-1.9.13.jar:/data/sysdir/hbase-1.1.2/lib/jackson-xc-1.9= .13.jar:/data/sysdir/hbase-1.1.2/lib/jamon-runtime-2.3.1.jar:/data/sysdir/h= base-1.1.2/lib/jasper-compiler-5.5.23.jar:/data/sysdir/hbase-1.1.2/lib/jasp= er-runtime-5.5.23.jar:/data/sysdir/hbase-1.1.2/lib/javax.inject-1.jar:/data= /sysdir/hbase-1.1.2/lib/java-xmlbuilder-0.4.jar:/data/sysdir/hbase-1.1.2/li= b/jaxb-api-2.2.2.jar:/data/sysdir/hbase-1.1.2/lib/jaxb-impl-2.2.3-1.jar:/da= ta/sysdir/hbase-1.1.2/lib/jcodings-1.0.8.jar:/data/sysdir/hbase-1.1.2/lib/j= ersey-client-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-core-1.9.jar:/data= /sysdir/hbase-1.1.2/lib/jersey-guice-1.9.jar:/data/sysdir/hbase-1.1.2/lib/j= ersey-json-1.9.jar:/data/sysdir/hbase-1.1.2/lib/jersey-server-1.9.jar:/data= /sysdir/hbase-1.1.2/lib/jets3t-0.9.0.jar:/data/sysdir/hbase-1.1.2/lib/jetti= son-1.3.3.jar:/data/sysdir/hbase-1.1.2/lib/jetty-6.1.26.jar:/data/sysdir/hb= ase-1.1.2/lib/jetty-sslengine-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/jetty= -util-6.1.26.jar:/data/sysdir/hbase-1.1.2/lib/joni-2.1.2.jar:/data/sysdir/h= base-1.1.2/lib/jruby-complete-1.6.8.jar:/data/sysdir/hbase-1.1.2/lib/jsch-0= .1.42.jar:/data/sysdir/hbase-1.1.2/lib/jsp-2.1-6.1.14.jar:/data/sysdir/hbas= e-1.1.2/lib/jsp-api-2.1-6.1.14.jar:/data/sysdir/hbase-1.1.2/lib/jsr305-1.3.= 9.jar:/data/sysdir/hbase-1.1.2/lib/junit-4.11.jar:/data/sysdir/hbase-1.1.2/= lib/leveldbjni-all-1.8.jar:/data/sysdir/hbase-1.1.2/lib/libthrift-0.9.0.jar= :/data/sysdir/hbase-1.1.2/lib/log4j-1.2.17.jar:/data/sysdir/hbase-1.1.2/lib= /metrics-core-2.2.0.jar:/data/sysdir/hbase-1.1.2/lib/netty-3.2.4.Final.jar:= /data/sysdir/hbase-1.1.2/lib/netty-all-4.0.23.Final.jar:/data/sysdir/hbase-= 1.1.2/lib/paranamer-2.3.jar:/data/sysdir/hbase-1.1.2/lib/protobuf-java-2.5.= 0.jar:/data/sysdir/hbase-1.1.2/lib/servlet-api-2.5-6.1.14.jar:/data/sysdir/= hbase-1.1.2/lib/servlet-api-2.5.jar:/data/sysdir/hbase-1.1.2/lib/slf4j-api-= 1.7.7.jar:/data/sysdir/hbase-1.1.2/lib/slf4j-log4j12-1.7.5.jar:/data/sysdir= /hbase-1.1.2/lib/snappy-java-1.0.4.1.jar:/data/sysdir/hbase-1.1.2/lib/spyme= mcached-2.11.6.jar:/data/sysdir/hbase-1.1.2/lib/xmlenc-0.52.jar:/data/sysdi= r/hbase-1.1.2/lib/xz-1.0.jar:/data/sysdir/hbase-1.1.2/lib/zookeeper-3.4.6.j= ar:/data/sysdir/hadoop-2.4.1-hbase/etc/hadoop:/data/sysdir/hadoop-2.4.1-hba= se/share/hadoop/common/lib/commons-net-3.1.jar:/data/sysdir/hadoop-2.4.1-hb= ase/share/hadoop/common/lib/jaxb-api-2.2.2.jar:/data/sysdir/hadoop-2.4.1-hb= ase/share/hadoop/common/lib/jets3t-0.9.0.jar:/data/sysdir/hadoop-2.4.1-hbas= e/share/hadoop/common/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbas= e/share/hadoop/common/lib/httpclient-4.2.5.jar:/data/sysdir/hadoop-2.4.1-hb= ase/share/hadoop/common/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-= 2.4.1-hbase/share/hadoop/common/lib/java-xmlbuilder-0.4.jar:/data/sysdir/ha= doop-2.4.1-hbase/share/hadoop/common/lib/asm-3.2.jar:/data/sysdir/hadoop-2.= 4.1-hbase/share/hadoop/common/lib/commons-collections-3.2.1.jar:/data/sysdi= r/hadoop-2.4.1-hbase/share/hadoop/common/lib/jettison-1.1.jar:/data/sysdir/= hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-logging-1.1.3.jar:/data/= sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-codec-1.4.jar:/da= ta/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-digester-1.8.j= ar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsr305-1.3.9.jar= :/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/activation-1.1.jar= :/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/xmlenc-0.52.jar:/d= ata/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-mapper-asl-1.= 8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jetty-6.1.2= 6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/log4j-1.2.17.= jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/netty-3.6.2.Fin= al.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/stax-api-1.0= -2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/hadoop-auth-= 2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jasper-co= mpiler-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/c= ommons-beanutils-core-1.8.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoo= p/common/lib/hadoop-annotations-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/s= hare/hadoop/common/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/= share/hadoop/common/lib/jasper-runtime-5.5.23.jar:/data/sysdir/hadoop-2.4.1= -hbase/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar:/data/sysdir/hadoop-= 2.4.1-hbase/share/hadoop/common/lib/avro-1.7.4.jar:/data/sysdir/hadoop-2.4.= 1-hbase/share/hadoop/common/lib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4= .1-hbase/share/hadoop/common/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/ha= doop-2.4.1-hbase/share/hadoop/common/lib/jetty-util-6.1.26.jar:/data/sysdir= /hadoop-2.4.1-hbase/share/hadoop/common/lib/jersey-server-1.9.jar:/data/sys= dir/hadoop-2.4.1-hbase/share/hadoop/common/lib/httpcore-4.2.5.jar:/data/sys= dir/hadoop-2.4.1-hbase/share/hadoop/common/lib/paranamer-2.3.jar:/data/sysd= ir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jackson-jaxrs-1.8.8.jar:/data= /sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/slf4j-api-1.7.5.jar:/dat= a/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/snappy-java-1.0.4.1.jar= :/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jsch-0.1.42.jar:/d= ata/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/guava-11.0.2.jar:/dat= a/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/protobuf-java-2.5.0.jar= :/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/commons-configurat= ion-1.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/mockito= -all-1.8.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jers= ey-json-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/jac= kson-xc-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/c= ommons-el-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/j= sp-api-2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/serv= let-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/lib/com= mons-httpclient-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common= /lib/zookeeper-3.4.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/commo= n/lib/commons-math3-3.1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= common/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoo= p/common/lib/junit-4.8.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/c= ommon/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common/li= b/jaxb-impl-2.2.3-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/common= /lib/commons-beanutils-1.7.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hado= op/common/hadoop-common-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/sha= re/hadoop/common/hadoop-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/sh= are/hadoop/common/hadoop-common-2.4.1-path-1.jar:/data/sysdir/hadoop-2.4.1-= hbase/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/data/sysdir/hadoo= p-2.4.1-hbase/share/hadoop/common/hadoop-nfs-2.4.1.jar:/data/sysdir/hadoop-= 2.4.1-hbase/share/hadoop/hdfs:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= hdfs/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hd= fs/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/co= mmons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/l= ib/commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/= lib/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/= xmlenc-0.52.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/jacks= on-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/l= ib/jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/l= og4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/netty= -3.6.2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/comm= ons-daemon-1.0.13.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib= /jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/lib/= jasper-runtime-5.5.23.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs= /lib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/= lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /hdfs/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoo= p/hdfs/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hado= op/hdfs/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/h= dfs/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoo= p/hdfs/lib/commons-el-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= hdfs/lib/jsp-api-2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/= lib/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/l= ib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/hdfs/h= adoop-hdfs-2.4.1-patch-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/h= dfs/hadoop-hdfs-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoo= p/hdfs/hadoop-hdfs-nfs-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hado= op/yarn/lib/jaxb-api-2.2.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/ha= doop/yarn/lib/javax.inject-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hado= op/yarn/lib/asm-3.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/l= ib/commons-collections-3.2.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hado= op/yarn/lib/jline-0.9.94.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/jettison-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/= lib/commons-logging-1.1.3.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= yarn/lib/commons-codec-1.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/jsr305-1.3.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/activation-1.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/= lib/jackson-mapper-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hado= op/yarn/lib/jetty-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/log4j-1.2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/= lib/leveldbjni-all-1.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yar= n/lib/stax-api-1.0-2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/= lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/l= ib/jersey-guice-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/l= ib/commons-cli-1.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/li= b/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/jetty-util-6.1.26.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= yarn/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/aopalliance-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= yarn/lib/jackson-jaxrs-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hado= op/yarn/lib/guice-servlet-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/had= oop/yarn/lib/guava-11.0.2.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/= yarn/lib/protobuf-java-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hado= op/yarn/lib/jersey-json-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoo= p/yarn/lib/jackson-xc-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoo= p/yarn/lib/servlet-api-2.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop= /yarn/lib/commons-httpclient-3.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/= hadoop/yarn/lib/zookeeper-3.4.5.jar:/data/sysdir/hadoop-2.4.1-hbase/share/h= adoop/yarn/lib/commons-lang-2.6.jar:/data/sysdir/hadoop-2.4.1-hbase/share/h= adoop/yarn/lib/guice-3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/y= arn/lib/xz-1.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/ja= xb-impl-2.2.3-1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/lib/j= ersey-client-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hado= op-yarn-server-applicationhistoryservice-2.4.1.jar:/data/sysdir/hadoop-2.4.= 1-hbase/share/hadoop/yarn/hadoop-yarn-applications-distributedshell-2.4.1.j= ar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-nod= emanager-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop= -yarn-server-resourcemanager-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/shar= e/hadoop/yarn/hadoop-yarn-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/= share/hadoop/yarn/hadoop-yarn-server-tests-2.4.1.jar:/data/sysdir/hadoop-2.= 4.1-hbase/share/hadoop/yarn/hadoop-yarn-client-2.4.1.jar:/data/sysdir/hadoo= p-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-server-web-proxy-2.4.1.jar:/dat= a/sysdir/hadoop-2.4.1-hbase/share/hadoop/yarn/hadoop-yarn-applications-unma= naged-am-launcher-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/ya= rn/hadoop-yarn-server-common-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/shar= e/hadoop/yarn/hadoop-yarn-api-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/sha= re/hadoop/mapreduce/lib/commons-io-2.4.jar:/data/sysdir/hadoop-2.4.1-hbase/= share/hadoop/mapreduce/lib/commons-compress-1.4.1.jar:/data/sysdir/hadoop-2= .4.1-hbase/share/hadoop/mapreduce/lib/hadoop-snappy-0.0.1-SNAPSHOT.jar:/dat= a/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/javax.inject-1.jar:/= data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/asm-3.2.jar:/data= /sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/jackson-mapper-asl-1.= 8.8.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/log4j-1.= 2.17.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/netty-3= .6.2.Final.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/h= adoop-annotations-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/ma= preduce/lib/jersey-core-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoo= p/mapreduce/lib/jersey-guice-1.9.jar:/data/sysdir/hadoop-2.4.1-hbase/share/= hadoop/mapreduce/lib/avro-1.7.4.jar:/data/sysdir/hadoop-2.4.1-hbase/share/h= adoop/mapreduce/lib/jackson-core-asl-1.8.8.jar:/data/sysdir/hadoop-2.4.1-hb= ase/share/hadoop/mapreduce/lib/jersey-server-1.9.jar:/data/sysdir/hadoop-2.= 4.1-hbase/share/hadoop/mapreduce/lib/paranamer-2.3.jar:/data/sysdir/hadoop-= 2.4.1-hbase/share/hadoop/mapreduce/lib/aopalliance-1.0.jar:/data/sysdir/had= oop-2.4.1-hbase/share/hadoop/mapreduce/lib/guice-servlet-3.0.jar:/data/sysd= ir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/d= ata/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/hamcrest-core-1.1.= jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/protobuf-jav= a-2.5.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/juni= t-4.10.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/guice= -3.0.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/lib/xz-1.0.= jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce= -examples-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/= hadoop-mapreduce-client-app-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share= /hadoop/mapreduce/hadoop-mapreduce-client-common-2.4.1.jar:/data/sysdir/had= oop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.= 4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapre= duce-client-core-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/map= reduce/hadoop-mapreduce-client-shuffle-2.4.1.jar:/data/sysdir/hadoop-2.4.1-= hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.4.1.jar:/data/sys= dir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-pl= ugins-2.4.1.jar:/data/sysdir/hadoop-2.4.1-hbase/share/hadoop/mapreduce/hado= op-mapreduce-client-jobclient-2.4.1-tests.jar:/data/sysdir/hadoop-2.4.1-hba= se/contrib/capacity-scheduler/*.jar > 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environm= ent:java.library.path=3D/data/sysdir/hadoop-2.4.1-hbase/lib/native > 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environm= ent:java.io.tmpdir=3D/tmp > 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environm= ent:java.compiler=3D > 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environm= ent:os.name=3DLinux > 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environm= ent:os.arch=3Damd64 > 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environm= ent:os.version=3D3.13.6 > 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environm= ent:user.name=3Dhbase > 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environm= ent:user.home=3D/data/home/hbase > 2016-08-14 22:17:03,178 INFO [main] zookeeper.ZooKeeper: Client environm= ent:user.dir=3D/data/sysdir/hbase-1.1.2/conf > 2016-08-14 22:17:03,180 INFO [main] zookeeper.ZooKeeper: Initiating clie= nt connection, connectString=3D192.168.201.52:2181,192.168.201.53:2181,192.= 168.201.63:2181,192.168.201.64:2181,192.168.201.65:2181 sessionTimeout=3D30= 000 watcher=3Dorg.apache.zookeeper.ZooKeeperMain$MyWatcher@38987d26 > 2016-08-14 22:17:03,206 INFO [main-SendThread(192.168.201.65:2181)] zook= eeper.ClientCnxn: Opening socket connection to server 192.168.201.65/192.16= 8.201.65:2181. Will not attempt to authenticate using SASL (unknown error) > 2016-08-14 22:17:03,228 INFO [main-SendThread(192.168.201.65:2181)] zook= eeper.ClientCnxn: Socket connection established to 192.168.201.65/192.168.2= 01.65:2181, initiating session > 2016-08-14 22:17:03,234 INFO [main-SendThread(192.168.201.65:2181)] zook= eeper.ClientCnxn: Session establishment complete on server 192.168.201.65/1= 92.168.201.65:2181, sessionid =3D 0x556683846220a79, negotiated timeout =3D= 30000 > {quote} > Then i looked into zk's log, i found these : > {quote} > 8/14/16 10:17:03 PM CST session 0x556683846220a79 cxid 0x0 zxid 0xf73ca58= 59 createSession 30000 > 8/14/16 10:17:03 PM CST session 0x556683846220a79 cxid 0x1 zxid 0xf73ca58= 5a delete '/olapHbase/rs/olap3.data.lq%2C16020%2C1470799848293 > {quote} > And i looked master logs as below :=20 > {quote} > 2016-08-14 22:16:43,942 INFO [ProcedureExecutorThread-0] zookeeper.ZKTab= leStateManager: Moving table KYLIN_F5O605Z3QZ state from null to ENABLING > 2016-08-14 22:16:43,952 INFO [ProcedureExecutorThread-0] master.Assignme= ntManager: Bulk assigning 1 region(s) across 4 server(s), round-robin=3Dtru= e > 2016-08-14 22:16:43,953 INFO [olap1.data.lq,16000,1469605579716-GeneralB= ulkAssigner-2] master.AssignmentManager: Assigning 1 region(s) to olap3.dat= a.lq,16020,1470799848293 > 2016-08-14 22:16:43,958 INFO [olap1.data.lq,16000,1469605579716-GeneralB= ulkAssigner-2] master.RegionStates: Transition {6e207f0307d4213d6c0e195115d= f1b8d state=3DOFFLINE, ts=3D1471184203953, server=3Dnull} to {6e207f0307d42= 13d6c0e195115df1b8d state=3DPENDING_OPEN, ts=3D1471184203958, server=3Dolap= 3.data.lq,16020,1470799848293} > 2016-08-14 22:16:43,985 INFO [AM.ZK.Worker-pool2-t4001] master.RegionSta= tes: Transition {6e207f0307d4213d6c0e195115df1b8d state=3DPENDING_OPEN, ts= =3D1471184203958, server=3Dolap3.data.lq,16020,1470799848293} to {6e207f030= 7d4213d6c0e195115df1b8d state=3DOPENING, ts=3D1471184203985, server=3Dolap3= .data.lq,16020,1470799848293} > 2016-08-14 22:17:03,238 INFO [main-EventThread] zookeeper.RegionServerTr= acker: RegionServer ephemeral node deleted, processing expiration [olap3.da= ta.lq,16020,1470799848293] > 2016-08-14 22:17:03,240 WARN [MASTER_SERVER_OPERATIONS-olap1:16000-2] hb= ase.HBaseConfiguration: Config option "hbase.regionserver.lease.period" is = deprecated. Instead, use "hbase.client.scanner.timeout.period" > 2016-08-14 22:17:04,008 INFO [ProcedureExecutorThread-0] master.Assignme= ntManager: Bulk assigning done > 2016-08-14 22:17:04,011 INFO [ProcedureExecutorThread-0] zookeeper.ZKTab= leStateManager: Moving table KYLIN_F5O605Z3QZ state from ENABLING to ENABLE= D > 2016-08-14 22:17:04,251 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] ha= ndler.ServerShutdownHandler: Splitting logs for olap3.data.lq,16020,1470799= 848293 before assignment; region count=3D2033 > 2016-08-14 22:17:04,261 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] ma= ster.SplitLogManager: dead splitlog workers [olap3.data.lq,16020,1470799848= 293] > 2016-08-14 22:17:04,263 INFO [MASTER_SERVER_OPERATIONS-olap1:16000-2] ma= ster.SplitLogManager: started splitting 2 logs in [hdfs://olapHbaseCluster:= 8020/hbase/WALs/olap3.data.lq,16020,1470799848293-splitting] for [olap3.dat= a.lq,16020,1470799848293] > 2016-08-14 22:17:04,268 INFO [main-EventThread] coordination.SplitLogMan= agerCoordination: task /olapHbase/splitWAL/WALs%2Folap3.data.lq%2C16020%2C1= 470799848293-splitting%2Folap3.data.lq%252C16020%252C1470799848293.default.= 1471182516751 acquired by olap1.data.lq,16020,1470799983323 > 2016-08-14 22:17:04,277 INFO [main-EventThread] coordination.SplitLogMan= agerCoordination: task /olapHbase/splitWAL/WALs%2Folap3.data.lq%2C16020%2C1= 470799848293-splitting%2Folap3.data.lq%252C16020%252C1470799848293.default.= 1471180125256 acquired by olap2.data.lq,16020,1470800075996 > 2016-08-14 22:17:04,288 WARN [olap1.data.lq,16000,1469605579716_ChoreSer= vice_1] hbase.HBaseConfiguration: Config option "hbase.regionserver.lease.p= eriod" is deprecated. Instead, use "hbase.client.scanner.timeout.period" > .... > {quote} > I'm not falimliar with hbase , may be it's not a bug , but i can not figu= re out why this happend. I will be grateful if somebody can help me. -- This message was sent by Atlassian JIRA (v6.3.4#6332)