Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
From: Lars George <lars.george@gmail.com>
Content-Type: text/plain;
	charset=utf-8
Content-Transfer-Encoding: quoted-printable
Mime-Version: 1.0 (1.0)
Subject: Re: HBase failing to restart in single-user mode
Message-Id: <B58195E0-7D40-4C25-B7C9-0D4DB6B2F4F1@gmail.com>
Date: Mon, 18 May 2015 07:39:46 +0200
References: 
 <CAFKYj4f+E5zdYuS2S5X2c-QR0tC-Ss8ZmV855GYYO5cwVUH23Q@mail.gmail.com>
In-Reply-To: 
 <CAFKYj4f+E5zdYuS2S5X2c-QR0tC-Ss8ZmV855GYYO5cwVUH23Q@mail.gmail.com>
To: "user@hbase.apache.org" <user@hbase.apache.org>

I noticed similar ZK related issues but those went away after changing the Z=
K directory to a permanent directory along with the HBase root directory. Bo=
th point now to a location in my home folder and restarts work fine now. Not=
 much help but wanted to at least state that. =20

Lars=20

Sent from my iPhone

> On 18 May 2015, at 05:55, tsuna <tsunanet@gmail.com> wrote:
>=20
> Hi all,
> For testing on my laptop (OSX with JDK 1.7.0_45) I usually build the
> latest version from branch-1.0 and use the following config:
>=20
> <configuration>
> <property>
>  <name>hbase.rootdir</name>
>  <value>file:///tmp/hbase-${user.name}</value>
> </property>
> <property>
>  <name>hbase.online.schema.update.enable</name>
>  <value>true</value>
> </property>
> <property>
>  <name>zookeeper.session.timeout</name>
>  <value>300000</value>
> </property>
> <property>
>  <name>hbase.zookeeper.property.tickTime</name>
>  <value>2000000</value>
> </property>
>  <property>
>    <name>hbase.zookeeper.dns.interface</name>
>    <value>lo0</value>
>  </property>
>  <property>
>    <name>hbase.regionserver.dns.interface</name>
>    <value>lo0</value>
>  </property>
>  <property>
>    <name>hbase.master.dns.interface</name>
>    <value>lo0</value>
>  </property>
> </configuration>
>=20
> Since at least a month ago (perhaps longer, I don=E2=80=99t remember exact=
ly)
> I can=E2=80=99t restart HBase.  The very first time it starts up fine, but=

> subsequent startup attempts all fail with:
>=20
> 2015-05-17 20:39:19,024 INFO  [RpcServer.responder] ipc.RpcServer:
> RpcServer.responder: starting
> 2015-05-17 20:39:19,024 INFO  [RpcServer.listener,port=3D49809]
> ipc.RpcServer: RpcServer.listener,port=3D49809: starting
> 2015-05-17 20:39:19,029 INFO  [main] http.HttpRequestLog: Http request
> log for http.requests.regionserver is not defined
> 2015-05-17 20:39:19,030 INFO  [main] http.HttpServer: Added global
> filter 'safety'
> (class=3Dorg.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)
> 2015-05-17 20:39:19,031 INFO  [main] http.HttpServer: Added filter
> static_user_filter
> (class=3Dorg.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFi=
lter)
> to context regionserver
> 2015-05-17 20:39:19,031 INFO  [main] http.HttpServer: Added filter
> static_user_filter
> (class=3Dorg.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFi=
lter)
> to context static
> 2015-05-17 20:39:19,031 INFO  [main] http.HttpServer: Added filter
> static_user_filter
> (class=3Dorg.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFi=
lter)
> to context logs
> 2015-05-17 20:39:19,033 INFO  [main] http.HttpServer: Jetty bound to port 4=
9811
> 2015-05-17 20:39:19,033 INFO  [main] mortbay.log: jetty-6.1.26
> 2015-05-17 20:39:19,157 INFO  [main] mortbay.log: Started
> SelectChannelConnector@0.0.0.0:49811
> 2015-05-17 20:39:19,222 INFO  [M:0;localhost:49807]
> zookeeper.RecoverableZooKeeper: Process
> identifier=3Dhconnection-0x4f708099 connecting to ZooKeeper
> ensemble=3Dlocalhost:2181
> 2015-05-17 20:39:19,222 INFO  [M:0;localhost:49807]
> zookeeper.ZooKeeper: Initiating client connection,
> connectString=3Dlocalhost:2181 sessionTimeout=3D10000
> watcher=3Dhconnection-0x4f7080990x0, quorum=3Dlocalhost:2181,
> baseZNode=3D/hbase
> 2015-05-17 20:39:19,223 INFO
> [M:0;localhost:49807-SendThread(localhost:2181)] zookeeper.ClientCnxn:
> Opening socket connection to server localhost/127.0.0.1:2181. Will not
> attempt to authenticate using SASL (unknown error)
> 2015-05-17 20:39:19,223 INFO
> [M:0;localhost:49807-SendThread(localhost:2181)] zookeeper.ClientCnxn:
> Socket connection established to localhost/127.0.0.1:2181, initiating
> session
> 2015-05-17 20:39:19,223 INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> server.NIOServerCnxnFactory: Accepted socket connection from
> /127.0.0.1:49812
> 2015-05-17 20:39:19,223 INFO
> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer:
> Client attempting to establish new session at /127.0.0.1:49812
> 2015-05-17 20:39:19,224 INFO  [SyncThread:0] server.ZooKeeperServer:
> Established session 0x14d651aaec00002 with negotiated timeout 4000000
> for client /127.0.0.1:49812
> 2015-05-17 20:39:19,224 INFO
> [M:0;localhost:49807-SendThread(localhost:2181)] zookeeper.ClientCnxn:
> Session establishment complete on server localhost/127.0.0.1:2181,
> sessionid =3D 0x14d651aaec00002, negotiated timeout =3D 4000000
> 2015-05-17 20:39:19,249 INFO  [M:0;localhost:49807]
> regionserver.HRegionServer: ClusterId :
> 6ad7eddd-2886-4ff0-b377-a2ff42c8632f
> 2015-05-17 20:39:49,208 ERROR [main] master.HMasterCommandLine: Master exi=
ting
> java.lang.RuntimeException: Master not active after 30 seconds
>        at org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUt=
il.java:194)
>        at org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseClus=
ter.java:445)
>        at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HM=
asterCommandLine.java:197)
>        at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCom=
mandLine.java:139)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerComm=
andLine.java:126)
>        at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2002)
>=20
>=20
> I noticed that this has something to do with the ZooKeeper data.  If I
> rm -rf $TMPDIR/hbase-tsuna/zookeeper then I can start HBase again.
> But of course HBase won=E2=80=99t work properly because while some tables
> exist on the filesystem, they no longer exist in ZK, etc.
>=20
> Does anybody know what could be left behind in ZK that could make it
> hang during startup?  I looked at a jstack output while it was paused
> during 30s and didn=E2=80=99t find anything noteworthy.
>=20
> --=20
> Benoit "tsuna" Sigoure