Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CAFKYj4c3nt1ofy5e8enDocLDNrEqgwLGKmxVYdO3asuEmS_fyg@mail.gmail.com>
References: 
 <CAFKYj4f+E5zdYuS2S5X2c-QR0tC-Ss8ZmV855GYYO5cwVUH23Q@mail.gmail.com>
 <B58195E0-7D40-4C25-B7C9-0D4DB6B2F4F1@gmail.com>
 <CALckxSOQSq7VVYh9bL9Y9xVyy8jSM4QK5Wk4RPP2TjYwd3stTA@mail.gmail.com>
 <CAFKYj4c3nt1ofy5e8enDocLDNrEqgwLGKmxVYdO3asuEmS_fyg@mail.gmail.com>
From: anil gupta <anilgupta84@gmail.com>
Date: Mon, 18 May 2015 21:52:25 -0700
Message-ID: 
 <CAF1+Vs9JZxWCiX05zWt8UcZ8WenUBA-NL-xaCbHKrqaRb59EFg@mail.gmail.com>
Subject: Re: HBase failing to restart in single-user mode
To: "user@hbase.apache.org" <user@hbase.apache.org>
Content-Type: multipart/alternative; boundary=001a11396180e134e0051668144b

--001a11396180e134e0051668144b
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Benoit,
I think you need to move the directory out of "/tmp" and give it a shot.
/tmp/hbase-${user.name}
/zk will get cleaned up during restart.


~Anil

On Mon, May 18, 2015 at 9:45 PM, tsuna <tsunanet@gmail.com> wrote:

> I added this to hbase-site.xml:
>
> <property>
>   <name>hbase.zookeeper.property.dataDir</name>
>   <value>/tmp/hbase-${user.name}/zk</value>
> </property>
>
> Didn=E2=80=99t change anything.  Once I kill/shutdown HBase, it won=E2=80=
=99t come back up.
>
> On Mon, May 18, 2015 at 1:14 AM, Viral Bajaria <viral.bajaria@gmail.com>
> wrote:
> > Same for me, I had faced similar issues especially on my virtual machin=
es
> > since I would restart them more often than my host machine.
> >
> > Moving ZK from /tmp which could get cleared on reboots fixed the issue
> for
> > me.
> >
> > Thanks,
> > Viral
> >
> >
> > On Sun, May 17, 2015 at 10:39 PM, Lars George <lars.george@gmail.com>
> wrote:
> >
> >> I noticed similar ZK related issues but those went away after changing
> the
> >> ZK directory to a permanent directory along with the HBase root
> directory.
> >> Both point now to a location in my home folder and restarts work fine
> now.
> >> Not much help but wanted to at least state that.
> >>
> >> Lars
> >>
> >> Sent from my iPhone
> >>
> >> > On 18 May 2015, at 05:55, tsuna <tsunanet@gmail.com> wrote:
> >> >
> >> > Hi all,
> >> > For testing on my laptop (OSX with JDK 1.7.0_45) I usually build the
> >> > latest version from branch-1.0 and use the following config:
> >> >
> >> > <configuration>
> >> > <property>
> >> >  <name>hbase.rootdir</name>
> >> >  <value>file:///tmp/hbase-${user.name}</value>
> >> > </property>
> >> > <property>
> >> >  <name>hbase.online.schema.update.enable</name>
> >> >  <value>true</value>
> >> > </property>
> >> > <property>
> >> >  <name>zookeeper.session.timeout</name>
> >> >  <value>300000</value>
> >> > </property>
> >> > <property>
> >> >  <name>hbase.zookeeper.property.tickTime</name>
> >> >  <value>2000000</value>
> >> > </property>
> >> >  <property>
> >> >    <name>hbase.zookeeper.dns.interface</name>
> >> >    <value>lo0</value>
> >> >  </property>
> >> >  <property>
> >> >    <name>hbase.regionserver.dns.interface</name>
> >> >    <value>lo0</value>
> >> >  </property>
> >> >  <property>
> >> >    <name>hbase.master.dns.interface</name>
> >> >    <value>lo0</value>
> >> >  </property>
> >> > </configuration>
> >> >
> >> > Since at least a month ago (perhaps longer, I don=E2=80=99t remember=
 exactly)
> >> > I can=E2=80=99t restart HBase.  The very first time it starts up fin=
e, but
> >> > subsequent startup attempts all fail with:
> >> >
> >> > 2015-05-17 20:39:19,024 INFO  [RpcServer.responder] ipc.RpcServer:
> >> > RpcServer.responder: starting
> >> > 2015-05-17 20:39:19,024 INFO  [RpcServer.listener,port=3D49809]
> >> > ipc.RpcServer: RpcServer.listener,port=3D49809: starting
> >> > 2015-05-17 20:39:19,029 INFO  [main] http.HttpRequestLog: Http reque=
st
> >> > log for http.requests.regionserver is not defined
> >> > 2015-05-17 20:39:19,030 INFO  [main] http.HttpServer: Added global
> >> > filter 'safety'
> >> > (class=3Dorg.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)
> >> > 2015-05-17 20:39:19,031 INFO  [main] http.HttpServer: Added filter
> >> > static_user_filter
> >> >
> >>
> (class=3Dorg.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserF=
ilter)
> >> > to context regionserver
> >> > 2015-05-17 20:39:19,031 INFO  [main] http.HttpServer: Added filter
> >> > static_user_filter
> >> >
> >>
> (class=3Dorg.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserF=
ilter)
> >> > to context static
> >> > 2015-05-17 20:39:19,031 INFO  [main] http.HttpServer: Added filter
> >> > static_user_filter
> >> >
> >>
> (class=3Dorg.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserF=
ilter)
> >> > to context logs
> >> > 2015-05-17 20:39:19,033 INFO  [main] http.HttpServer: Jetty bound to
> >> port 49811
> >> > 2015-05-17 20:39:19,033 INFO  [main] mortbay.log: jetty-6.1.26
> >> > 2015-05-17 20:39:19,157 INFO  [main] mortbay.log: Started
> >> > SelectChannelConnector@0.0.0.0:49811
> >> > 2015-05-17 20:39:19,222 INFO  [M:0;localhost:49807]
> >> > zookeeper.RecoverableZooKeeper: Process
> >> > identifier=3Dhconnection-0x4f708099 connecting to ZooKeeper
> >> > ensemble=3Dlocalhost:2181
> >> > 2015-05-17 20:39:19,222 INFO  [M:0;localhost:49807]
> >> > zookeeper.ZooKeeper: Initiating client connection,
> >> > connectString=3Dlocalhost:2181 sessionTimeout=3D10000
> >> > watcher=3Dhconnection-0x4f7080990x0, quorum=3Dlocalhost:2181,
> >> > baseZNode=3D/hbase
> >> > 2015-05-17 20:39:19,223 INFO
> >> > [M:0;localhost:49807-SendThread(localhost:2181)] zookeeper.ClientCnx=
n:
> >> > Opening socket connection to server localhost/127.0.0.1:2181. Will
> not
> >> > attempt to authenticate using SASL (unknown error)
> >> > 2015-05-17 20:39:19,223 INFO
> >> > [M:0;localhost:49807-SendThread(localhost:2181)] zookeeper.ClientCnx=
n:
> >> > Socket connection established to localhost/127.0.0.1:2181, initiatin=
g
> >> > session
> >> > 2015-05-17 20:39:19,223 INFO
> >> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181]
> >> > server.NIOServerCnxnFactory: Accepted socket connection from
> >> > /127.0.0.1:49812
> >> > 2015-05-17 20:39:19,223 INFO
> >> > [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181] server.ZooKeeperServer:
> >> > Client attempting to establish new session at /127.0.0.1:49812
> >> > 2015-05-17 20:39:19,224 INFO  [SyncThread:0] server.ZooKeeperServer:
> >> > Established session 0x14d651aaec00002 with negotiated timeout 400000=
0
> >> > for client /127.0.0.1:49812
> >> > 2015-05-17 20:39:19,224 INFO
> >> > [M:0;localhost:49807-SendThread(localhost:2181)] zookeeper.ClientCnx=
n:
> >> > Session establishment complete on server localhost/127.0.0.1:2181,
> >> > sessionid =3D 0x14d651aaec00002, negotiated timeout =3D 4000000
> >> > 2015-05-17 20:39:19,249 INFO  [M:0;localhost:49807]
> >> > regionserver.HRegionServer: ClusterId :
> >> > 6ad7eddd-2886-4ff0-b377-a2ff42c8632f
> >> > 2015-05-17 20:39:49,208 ERROR [main] master.HMasterCommandLine: Mast=
er
> >> exiting
> >> > java.lang.RuntimeException: Master not active after 30 seconds
> >> >        at
> >>
> org.apache.hadoop.hbase.util.JVMClusterUtil.startup(JVMClusterUtil.java:1=
94)
> >> >        at
> >>
> org.apache.hadoop.hbase.LocalHBaseCluster.startup(LocalHBaseCluster.java:=
445)
> >> >        at
> >>
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterComm=
andLine.java:197)
> >> >        at
> >>
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.=
java:139)
> >> >        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >> >        at
> >>
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.j=
ava:126)
> >> >        at
> org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2002)
> >> >
> >> >
> >> > I noticed that this has something to do with the ZooKeeper data.  If=
 I
> >> > rm -rf $TMPDIR/hbase-tsuna/zookeeper then I can start HBase again.
> >> > But of course HBase won=E2=80=99t work properly because while some t=
ables
> >> > exist on the filesystem, they no longer exist in ZK, etc.
> >> >
> >> > Does anybody know what could be left behind in ZK that could make it
> >> > hang during startup?  I looked at a jstack output while it was pause=
d
> >> > during 30s and didn=E2=80=99t find anything noteworthy.
> >> >
> >> > --
> >> > Benoit "tsuna" Sigoure
> >>
>
>
>
> --
> Benoit "tsuna" Sigoure
>


--=20
Thanks & Regards,
Anil Gupta

--001a11396180e134e0051668144b--