Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-RocketYMMF:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type;
  b=XdrvR+4nunbAJxPNh1oen7sZWMz+i5Aj9T0rYFJd8KptgMI2Z7MW0tjlZkmyCqZAhmPloYUK6jtmKGxppcJtY4HpEh2OPGaQ9u59WdBbWSobVemZP5g+CsaBE+TDXo+ozeXp2JM8C9MJYsYR9ThnTvm/nx3lCVd2iCXTckOLetk=;
References: <1363223561.19602.YahooMailNeo@web140606.mail.bf1.yahoo.com>
 <CALte62xeoqjoaCRoB=eLqEt+LrO6Vb+8CP=cKpd7GLirGNR7fg@mail.gmail.com>
 <1363224475.25762.YahooMailNeo@web140604.mail.bf1.yahoo.com>
 <CAFLnt_o69XtDVWthRBWraszaB7fCTFdq+jSe2em9VFsn+fPbRA@mail.gmail.com>
Message-ID: <1363225503.81869.YahooMailNeo@web140606.mail.bf1.yahoo.com>
Date: Wed, 13 Mar 2013 18:45:03 -0700 (PDT)
From: lars hofhansl <larsh@apache.org>
Reply-To: lars hofhansl <larsh@apache.org>
Subject: Re: Replication hosed after simple cluster restart
To: Himanshu Vashishtha <hvashish@cs.ualberta.ca>,
  "dev@hbase.apache.org" <dev@hbase.apache.org>
Cc: Ted Yu <yuzhihong@gmail.com>
In-Reply-To: 
 <CAFLnt_o69XtDVWthRBWraszaB7fCTFdq+jSe2em9VFsn+fPbRA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="1905101558-1863569917-1363225503=:81869"

--1905101558-1863569917-1363225503=:81869
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

Hey no problem. It's cool that we found it in a test env. It's probably qui=
te hard to reproduce.=0AThis is in 0.94.5 but this feature is off by defaul=
t.=0A=0AWhat's the general thought here, should I kill the current 0.94.6 r=
c for this?=0AMy gut says: Yes.=0A=0A=0A=0AI'm also a bit worried about the=
se:=0A2013-03-14 01:42:42,271 DEBUG org.apache.hadoop.hbase.replication.reg=
ionserver.ReplicationSource: Opening log for replication shared-dnds1-12-sf=
m.ops.sfdc.net%2C60020%2C1363220608780.1363220609572 at 0=0A2013-03-14 01:4=
2:42,358 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationS=
ource: 1 Got: =0Ajava.io.EOFException=0A=A0=A0=A0=A0=A0=A0=A0 at java.io.Da=
taInputStream.readFully(DataInputStream.java:180)=0A=A0=A0=A0=A0=A0=A0=A0 a=
t java.io.DataInputStream.readFully(DataInputStream.java:152)=0A=A0=A0=A0=
=A0=A0=A0=A0 at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.=
java:1800)=0A=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.io.SequenceFile$Rea=
der.initialize(SequenceFile.java:1765)=0A=A0=A0=A0=A0=A0=A0=A0 at org.apach=
e.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)=0A=A0=A0=A0=
=A0=A0=A0=A0 at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFil=
e.java:1728)=0A=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hbase.regionserve=
r.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)=
=0A=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hbase.regionserver.wal.Sequen=
ceFileLogReader.init(SequenceFileLogReader.java:177)=0A=A0=A0=A0=A0=A0=A0=
=A0 at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:72=
8)=0A=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hbase.replication.regionser=
ver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.ja=
va:67)=0A=A0=A0=A0=A0=A0=A0=A0 at org.apache.hadoop.hbase.replication.regio=
nserver.ReplicationSource.openReader(ReplicationSource.java:507)=0A=A0=A0=
=A0=A0=A0=A0=A0 at org.apache.hadoop.hbase.replication.regionserver.Replica=
tionSource.run(ReplicationSource.java:313)=0A2013-03-14 01:42:42,358 WARN o=
rg.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Waited t=
oo long for this file, considering dumping=0A2013-03-14 01:42:42,358 DEBUG =
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unable =
to open a reader, sleeping 1000 times 10=0A=0A=0AThis happens after bouncin=
g the cluster a 2nd time and these messages repeat every 10s (for hours now=
). This is a separate problem I think.=0A=0A=0A-- Lars=0A=0A=0A=0A_________=
_______________________=0A From: Himanshu Vashishtha <hvashish@cs.ualberta.=
ca>=0ATo: dev@hbase.apache.org; lars hofhansl <larsh@apache.org> =0ACc: Ted=
 Yu <yuzhihong@gmail.com> =0ASent: Wednesday, March 13, 2013 6:38 PM=0ASubj=
ect: Re: Replication hosed after simple cluster restart=0A =0AThis is bad. =
Yes, copyQueuesFromRSUsingMulti returns a list which it=0Amight not be able=
 to move later on, resulting in bogus znodes.=0AI'll fix this asap. Weird i=
t didn't happen in my testing earlier.=0ASorry about this.=0A=0AOn Wed, Mar=
 13, 2013 at 6:27 PM, lars hofhansl <larsh@apache.org> wrote:=0A> Sorry 0.9=
4.6RC1=0A> (I complain about folks not reporting the version all the time, =
and then I do it too)=0A>=0A>=0A>=0A> ________________________________=0A>=
=A0 From: Ted Yu <yuzhihong@gmail.com>=0A> To: dev@hbase.apache.org; lars h=
ofhansl <larsh@apache.org>=0A> Sent: Wednesday, March 13, 2013 6:17 PM=0A> =
Subject: Re: Replication hosed after simple cluster restart=0A>=0A>=0A> Did=
 this happen on 0.94.5 ?=0A>=0A> Thanks=0A>=0A>=0A> On Wed, Mar 13, 2013 at=
 6:12 PM, lars hofhansl <larsh@apache.org> wrote:=0A>=0A> We just ran into =
an interesting scenario. We restarted a cluster that was setup as a replica=
tion source.=0A>>The stop went cleanly.=0A>>=0A>>Upon restart *all* regions=
ervers aborted within a few seconds with variations of these errors:=0A>>ht=
tp://pastebin.com/3iQVuBqS=0A>>=0A>>This is scary!=0A>>=0A>>-- Lars
--1905101558-1863569917-1363225503=:81869--