Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-RocketYMMF:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=Sae7C6Df/DgdEpCcmtXhqI/ucC033SiVB4MhNDl3JSCaNwiD2k6vax/hATJkRiv9gEl8bQRjlv3Ng/7dIPSEEu8+GNhdU62JFebZMwI6gWGzHDA1QWQTgvMFJGRtDWUwmoDh7i12NRGy+RVtE1If1wHL10raXRCkdaY6LMArGHM=;
References: <1368081563.24364.YahooMailNeo@web140605.mail.bf1.yahoo.com>
 <1368084192.71836.YahooMailNeo@web140601.mail.bf1.yahoo.com>
 <1368085284.5783.YahooMailNeo@web140606.mail.bf1.yahoo.com>
 <D4C45A51-F18C-475E-8BD3-03D8BD12D335@gmail.com>
 <CA+RK=_BmoCJEtOt99Zd9sVdCARpS+L8-6gQpBfRqgZOw9679WA@mail.gmail.com>
 <F9F1914C-93ED-425A-85B3-2AAF138AC33F@gmail.com>
 <1368114531.12947.YahooMailNeo@web140604.mail.bf1.yahoo.com>
 <CALte62zeFan8nEPz9xuGBOjfUZZww=3Xr7GQ0nXpnQUsbC1fWg@mail.gmail.com>
 <1368116196.57721.YahooMailNeo@web140605.mail.bf1.yahoo.com>
 <CAKxWWm15in38zrfcPQEchgh9CyU-XK7+E0U7AZ3oGdHg7c56XQ@mail.gmail.com>
 <CAKxWWm3-zRrcFg+2+txj=icq7-T9WbMbPmKCMucTVNDC4WPXuw@mail.gmail.com>
 <CALte62wb4_Me2MQVH_pZWFd1meG-BN68xVJ9vHJ3x8jH156peg@mail.gmail.com>
 <1368119020.84034.YahooMailNeo@web140604.mail.bf1.yahoo.com>
 <CADcMMgHqVahZSgggfsSPrphEZiK=7M+o0-NLSKPHSy291jchfQ@mail.gmail.com>
 <1368123230.50177.YahooMailNeo@web140602.mail.bf1.yahoo.com>
 <1368124118.35096.YahooMailNeo@web140603.mail.bf1.yahoo.com>
 <CAMUu0w9HpaZsdA4QXODM8XF3aKRydH_-GCGf8eOshqfV9bNDHA@mail.gmail.com>
 <1368159953.93614.YahooMailNeo@web140603.mail.bf1.yahoo.com>
 <CAMUu0w9Nqir1r4zCKb3cU3QxyBmw6t2w-No4Zm6Fq7wU4E-G4w@mail.gmail.com>
Message-ID: <1368164834.16359.YahooMailNeo@web140606.mail.bf1.yahoo.com>
Date: Thu, 9 May 2013 22:47:14 -0700 (PDT)
From: lars hofhansl <larsh@apache.org>
Reply-To: lars hofhansl <larsh@apache.org>
Subject: Re: All region server died due to "Parent directory doesn't exist"
To: "dev@hbase.apache.org" <dev@hbase.apache.org>
In-Reply-To: 
 <CAMUu0w9Nqir1r4zCKb3cU3QxyBmw6t2w-No4Zm6Fq7wU4E-G4w@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="1905101558-283183563-1368164834=:16359"

--1905101558-283183563-1368164834=:16359
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Nope. That does not appear to be the problem.=0A=0A=0A_____________________=
___________=0A From: Enis S=F6ztutar <enis.soz@gmail.com>=0ATo: "dev@hbase.=
apache.org" <dev@hbase.apache.org>; lars hofhansl <larsh@apache.org> =0ASen=
t: Thursday, May 9, 2013 10:01 PM=0ASubject: Re: All region server died due=
 to "Parent directory doesn't exist"=0A =0A=0ABut you see the zookeeper ses=
sion timeout events in RS logs, and the master=0Asays that zk session for t=
he RS's has expired, right?=0A=0A=0AOn Thu, May 9, 2013 at 9:25 PM, lars ho=
fhansl <larsh@apache.org> wrote:=0A=0A> Still looking. Stack and Himanshu a=
re looking too (tanks again!).=0A>=0A> What I do know is that it has to do =
the fencing mechanism during log=0A> splitting.=0A> Until I bounced HDFS an=
d ZK (ZK probably being the culprit) each started=0A> RegionServer would im=
mediately be fenced off (it's log directory renamed).=0A> Probably by the S=
SH.=0A>=0A> It is not clear what caused the first RS to die. While there is=
 no direct=0A> evidence, from the logs it looks like the log directory was =
just suddenly=0A> renamed.=0A>=0A> I'll spend more time in the logs and als=
o watch for this happening again.=0A>=0A> We did find another misconfigured=
 cluster that had some services pointed=0A> at this cluster. It does not lo=
ok like that was actually a problem - there=0A> is no evidence in the logs =
that this actually caused a problem, but it made=0A> this deploy somewhat "=
special".=0A>=0A>=0A> -- Lars=0A>=0A>=0A>=0A> _____________________________=
___=0A>=A0 From: Enis S=F6ztutar <enis.soz@gmail.com>=0A> To: "dev@hbase.ap=
ache.org" <dev@hbase.apache.org>; lars hofhansl <=0A> larsh@apache.org>=0A>=
 Sent: Thursday, May 9, 2013 6:10 PM=0A> Subject: Re: All region server die=
d due to "Parent directory doesn't exist"=0A>=0A>=0A>=0A> Could we able to =
find the root cause?=0A>=0A>=0A>=0A> On Thu, May 9, 2013 at 11:28 AM, lars =
hofhansl <larsh@apache.org> wrote:=0A>=0A> Good news is that as far as I ca=
n tell no data was lost.=0A> >Eventually all logs were split and replayed.=
=0A> >=0A> >=0A> >=0A> >-- Lars=0A> >=0A> >=0A> >=0A> >----- Original Messa=
ge -----=0A> >=0A> >From: lars hofhansl <larsh@apache.org>=0A> >To: HBase D=
ev List <dev@hbase.apache.org>=0A> >=0A> >Cc:=0A> >Sent: Thursday, May 9, 2=
013 11:13 AM=0A> >Subject: Re: All region server died due to "Parent direct=
ory doesn't=0A> exist"=0A> >=0A> >Thanks Stack.=0A> >=0A> >I sent the logs.=
=0A> >Also, I have since bounced HDFS and ZK and the problem is gone now (I=
 can=0A> start RSs again and they stay up). Something got into a weird stat=
e.=0A> >=0A> >=0A> >-- Lars=0A> >=0A> >=0A> >=0A> >________________________=
________=0A> >From: Stack <stack@duboce.net>=0A> >To: HBase Dev List <dev@h=
base.apache.org>; lars hofhansl <=0A> larsh@apache.org>=0A> >Sent: Thursday=
, May 9, 2013 10:34 AM=0A> >Subject: Re: All region server died due to "Par=
ent directory doesn't=0A> exist"=0A> >=0A> >=0A> >=0A> >Want to send me a r=
egionserver log Lars? (off-list)=0A> >St.Ack=0A> >=0A> >=0A> >=0A> >On Thu,=
 May 9, 2013 at 10:03 AM, lars hofhansl <larsh@apache.org> wrote:=0A> >=0A>=
 >Thanks Ted and Varun.=0A> >>=0A> >>=0A> >>Let me check on the .META. serv=
er.=0A> >>=0A> >>=0A> >>The majority (13) of the RSs died within 2 minutes.=
 The remaining 3 died=0A> over the following 10 minutes.=0A> >>So that woul=
d point to general issue. I did not see any ZK issues but=0A> I'll double c=
heck.=0A> >>=0A> >>=0A> >>It is just interesting that even now, if I start =
and RS it aborts within=0A> a minute or two, because of this issue.=0A> >>=
=0A> >>=0A> >>-- Lars=0A> >>=0A> >>=0A> >>----- Original Message -----=0A> =
>>From: Ted Yu <yuzhihong@gmail.com>=0A> >>To: dev@hbase.apache.org=0A> >>=
=0A> >>Cc:=0A> >>Sent: Thursday, May 9, 2013 9:51 AM=0A> >>Subject: Re: All=
 region server died due to "Parent directory doesn't=0A> exist"=0A> >>=0A> =
>>Thanks Varun for sharing your experience.=0A> >>=0A> >>Lars:=0A> >>Was th=
e server carrying .META. functioning properly around the time when=0A> >>yo=
u observed the problem ?=0A> >>=0A> >>Cheers=0A> >>=0A> >>On Thu, May 9, 20=
13 at 9:41 AM, Varun Sharma <varun@pinterest.com>=0A> wrote:=0A> >>=0A> >>>=
 I meant no NTP/clock synchronization b/w zookeeper quorum and the HBase=0A=
> >>> cluster. I am not sure if you are seeing the exact same issue though.=
=0A> We=0A> >>> did not have mass failures at the same time due to this..=
=0A> >>>=0A> >>> Thanks=0A> >>> Varun=0A> >>>=0A> >>>=0A> >>> On Thu, May 9=
, 2013 at 9:39 AM, Varun Sharma <varun@pinterest.com>=0A> wrote:=0A> >>>=0A=
> >>> > Btw, I am not 100 % sure but I have some seen something like this=
=0A> before:=0A> >>> >=0A> >>> > 1) ZK connection flakiness causes ephemera=
l nodes to expire=0A> >>> > 2) Master detects failure and renames the logs =
into a splitting=0A> directory=0A> >>> > - this is intentional so that in c=
ase that region server comes back=0A> up,=0A> >>> it=0A> >>> > cannot write=
 to the logs being split=0A> >>> > 3) Region server dies because the log is=
 renamed=0A> >>> >=0A> >>> > So, the yanking away of files is done by the H=
Base master and is=0A> expected=0A> >>> > if the master feels the server is=
 dead. We found that the Region=0A> server=0A> >>> > logs DFS exceptions li=
ke crazy (1000s of them) in that case and we=0A> always=0A> >>> > suspected=
 that this is some kind of DFS error but when we really go=0A> upto=0A> >>>=
 > the point where it started, we found some zookeeper session issues.=0A> =
>>> >=0A> >>> > We had two cases of this - either super high load or NTP/no=
 clock=0A> >>> > synchronization b/w the clusters causing this issue for us=
.=0A> >>> >=0A> >>> > Thanks=0A> >>> > Varun=0A> >>> >=0A> >>> >=0A> >>> > =
On Thu, May 9, 2013 at 9:16 AM, lars hofhansl <larsh@apache.org>=0A> wrote:=
=0A> >>> >=0A> >>> >> Thanks Ted. I'll do the same.=0A> >>> >>=0A> >>> >>=
=0A> >>> >> ----- Original Message -----=0A> >>> >> From: Ted Yu <yuzhihong=
@gmail.com>=0A> >>> >> To: dev@hbase.apache.org; lars hofhansl <larsh@apach=
e.org>=0A> >>> >> Cc:=0A> >>> >> Sent: Thursday, May 9, 2013 9:07 AM=0A> >>=
> >> Subject: Re: All region server died due to "Parent directory doesn't=
=0A> >>> >> exist"=0A> >>> >>=0A> >>> >> I went through the patch for HBASE=
-7824 one more time and didn't=0A> find=0A> >>> >> direct correlation to th=
e issue Lars reported.=0A> >>> >>=0A> >>> >> I am going over the other JIRA=
s in Lars' list.=0A> >>> >>=0A> >>> >> Cheers=0A> >>> >>=0A> >>> >> On Thu,=
 May 9, 2013 at 8:48 AM, lars hofhansl <larsh@apache.org>=0A> wrote:=0A> >>=
> >>=0A> >>> >> > I will try. I do not think this is the issue, though.=0A>=
 >>> >> >=0A> >>> >> > The master is up in my case.=0A> >>> >> > Right now =
the cluster is in a state where each region server=0A> aborts=0A> >>> >> it=
self=0A> >>> >> > shortly after being started (which coincides with having =
it's log=0A> >>> >> directory=0A> >>> >> > renamed to ...-splitting).=0A> >=
>> >> >=0A> >>> >> >=0A> >>> >> > This is a test cluster and I could just s=
tart from scratch... This=0A> >>> >> appears=0A> >>> >> > to be a serious e=
nough problem, though, and I would like to track=0A> down=0A> >>> >> the=0A=
> >>> >> > issue.=0A> >>> >> >=0A> >>> >> > -- Lars=0A> >>> >> >=0A> >>> >>=
 >=0A> >>> >> >=0A> >>> >> > ----- Original Message -----=0A> >>> >> > From=
: Ted Yu <yuzhihong@gmail.com>=0A> >>> >> > To: "dev@hbase.apache.org" <dev=
@hbase.apache.org>=0A> >>> >> > Cc: "dev@hbase.apache.org" <dev@hbase.apach=
e.org>=0A> >>> >> > Sent: Thursday, May 9, 2013 2:04 AM=0A> >>> >> > Subjec=
t: Re: All region server died due to "Parent directory=0A> doesn't=0A> >>> =
>> exist"=0A> >>> >> >=0A> >>> >> > The config came from hbase-7824.=0A> >>=
> >> >=0A> >>> >> > There are other JIRAs in Lars' list which are related t=
o log=0A> >>> splitting.=0A> >>> >> >=0A> >>> >> > I think more investigati=
on is needed.=0A> >>> >> >=0A> >>> >> > Cheers=0A> >>> >> >=0A> >>> >> > On=
 May 9, 2013, at 1:59 AM, Andrew Purtell <apurtell@apache.org>=0A> >>> wrot=
e:=0A> >>> >> >=0A> >>> >> > > So that is HBASE-7824, right?=0A> >>> >> > >=
=0A> >>> >> > > On Thu, May 9, 2013 at 4:33 PM, Ted Yu <yuzhihong@gmail.com=
>=0A> wrote:=0A> >>> >> > >=0A> >>> >> > >> hbase.master.wait.for.log.split=
ting=0A> >>> >> > >=0A> >>> >> > >=0A> >>> >> > >=0A> >>> >> > >=0A> >>> >>=
 > > --=0A> >>> >> > > Best regards,=0A> >>> >> > >=0A> >>> >> > >=A0  - An=
dy=0A> >>> >> > >=0A> >>> >> > > Problems worthy of attack prove their wort=
h by hitting back. -=0A> Piet=0A> >>> >> Hein=0A> >>> >> > > (via Tom White=
)=0A> >>> >> >=0A> >>> >> >=0A> >>> >>=0A> >>> >>=0A> >>> >=0A> >>>=0A> >>=
=0A> >>=0A> >=0A>
--1905101558-283183563-1368164834=:16359--