Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=X-YMail-OSG:Received:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=lbAnBut2ZkmFIdyrS8BkJSuvFjCq02Fi59OENQtghgwB+55DQLjyk1io7PgsR6LfxEYDF0H1mIoaBYBQxinxzGIoQ7FHdGt2SkudHru3pcBrPZ5gEwa6GWTLG6fYZsvJZrqdKFWNs1jMMYNofsFOlhvAd1B29KxR6p5USLRb6LE=;
References: 
 <CAB7cnggHneqBWK8KqRntA2nZuUvJrzVesNuKjWAtmKCE2LefzA@mail.gmail.com>
 <1347838400.96816.YahooMailNeo@web121706.mail.ne1.yahoo.com>
 <CAB7cnggXZdUr1p4yvxY=D4S=G1o-BDqgYOuiOkTsz1RASQGpiw@mail.gmail.com>
Message-ID: <1347854767.97374.YahooMailNeo@web121706.mail.ne1.yahoo.com>
Date: Sun, 16 Sep 2012 21:06:07 -0700 (PDT)
From: lars hofhansl <lhofhansl@yahoo.com>
Reply-To: lars hofhansl <lhofhansl@yahoo.com>
Subject: Re: java.io.IOEcxeption key k1 followed by a smaller key k2
To: "user@hbase.apache.org" <user@hbase.apache.org>
In-Reply-To: 
 <CAB7cnggXZdUr1p4yvxY=D4S=G1o-BDqgYOuiOkTsz1RASQGpiw@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
 boundary="-730330825-1954445335-1347854767=:97374"

---730330825-1954445335-1347854767=:97374
Content-Type: text/plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable

It would be good track this down. Any way you can share the tools you use t=
o load the data?=0AIs it easy for you to reproduce this problem?=0A=0AIt's =
possible (but not likely) that there is a bug in Hadoop 1.0. You should use=
 1.0.3.=0A=0A-- Lars=0A=0A=0A=0A________________________________=0A From: M=
ohamed Ibrahim <mibrahim@mibrahim.net>=0ATo: user@hbase.apache.org; lars ho=
fhansl <lhofhansl@yahoo.com>; Stack <stack@duboce.net> =0ASent: Sunday, Sep=
tember 16, 2012 7:13 PM=0ASubject: Re: java.io.IOEcxeption key k1 followed =
by a smaller key k2=0A =0A=0AHello Lars / Stack,=0A=0AThank you for respond=
ing.=0A=0AThe date on the files is March 9th 2012. It's been up since then,=
 I restarted hbase and hadoop once. I only have a single node that I'm runn=
ing my tests on. I'm currently running 0.92.1 on hadoop 1.0 . I hope I'm us=
ing the correct mix.=0A=0AI'm not using any external tools other than hbase=
 java api, and inspecting the data using the shell. I ran my program, the f=
ollowing day I found the stack dump on the console. I checked by scanning t=
he table that had the exception for the smaller key k1, with limit 2 from t=
he shell and no exceptions were thrown. I can also see that the following k=
ey is larger than the first one, so nothing is wrong.=0A=0AI faced the same=
 exception before and it happened more frequently when I did a lot of Puts =
and Deletes using Htable.batch . Essentially I was updating several inverte=
d indexes each in its own table on the data of rows when they get inserted,=
 and instead of doing single Deletes & Puts on the indexes I used batch. ba=
tch improved the performance but threw this exception more. I stopped using=
 batch and now doing single Puts & Deletes. The exception rarely gets throw=
n, but still sporadically gets thrown.=0A=0AI will read about the hfile too=
l, thanks for the pointers. I will also try figure out a set of steps that =
would repeat the exception so it is more helpful. I will also try 0.94.1 wi=
th batch and see if it will happen again, and will let you know and will fi=
le a bug if I can repeat it consistently.=0A=0AThank you,=0AMohamed Ibrahim=
=0A=0A=0AOn Sun, Sep 16, 2012 at 7:33 PM, lars hofhansl <lhofhansl@yahoo.co=
m> wrote:=0A=0AHmm... HBASE-6579 gets rid of that check, because we thought=
 it no longer necessary.=0A>No do you remember what you did leading up to t=
his?=0A>Did you write these HFiles with some other tool? Done some bulk imp=
ort, etc?=0A>=0A>=0A>-- Lars=0A>=0A>=0A>=0A>_______________________________=
_=0A>=A0From: Mohamed Ibrahim <m0brhm@gmail.com>=0A>To: user@hbase.apache.o=
rg=0A>Sent: Sunday, September 16, 2012 5:59 AM=0A>Subject: java.io.IOEcxept=
ion key k1 followed by a smaller key k2=0A>=0A>=0A>Hello All,=0A>=0A>I am u=
sing hbase 0.92.1 on hadoop 1 . I am getting those exceptions, and it=0A>se=
ems to me that it means that the hbase file is not sorted in order. So=0A>w=
hen the scanner goes through it, it finds a smaller key after its current.=
=0A>=0A>Is that related to https://issues.apache.org/jira/browse/HBASE-6579=
 ??=0A>=0A>It looks like upgrading to 0.94.1 (current stable) won't fix the=
 issue. Any=0A>recommendations ??=0A>=0A>Here is the stack dump:=0A>=A0 =A0=
 =A0 =A0 at=0A>org.apache.hadoop.hbase.regionserver.StoreScanner.next(Store=
Scanner.java:266)=0A>=A0 =A0 =A0 =A0 at=0A>org.apache.hadoop.hbase.regionse=
rver.KeyValueHeap.next(KeyValueHeap.java:138)=0A>=A0 =A0 =A0 =A0 at=0A>org.=
apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRe=
gion.java:2945)=0A>=A0 =A0 =A0 =A0 at=0A>org.apache.hadoop.hbase.regionserv=
er.HRegion$RegionScannerImpl.next(HRegion.java:2901)=0A>=A0 =A0 =A0 =A0 at=
=0A>org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRe=
gion.java:2918)=0A>=A0 =A0 =A0 =A0 at=0A>org.apache.hadoop.hbase.regionserv=
er.HRegion.getLastIncrement(HRegion.java:3660)=0A>=A0 =A0 =A0 =A0 at=0A>org=
.apache.hadoop.hbase.regionserver.HRegion.incrementColumnValue(HRegion.java=
:3844)=0A>=A0 =A0 =A0 =A0 at=0A>org.apache.hadoop.hbase.regionserver.HRegio=
nServer.incrementColumnValue(HRegionServer.java:2982)=0A>=A0 =A0 =A0 =A0 at=
 sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)=0A>=A0 =A0 =
=A0 =A0 at=0A>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMet=
hodAccessorImpl.java:43)=0A>=A0 =A0 =A0 =A0 at java.lang.reflect.Method.inv=
oke(Method.java:616)=0A>=A0 =A0 =A0 =A0 at=0A>org.apache.hadoop.hbase.ipc.W=
ritableRpcEngine$Server.call(WritableRpcEngine.java:364)=0A>=A0 =A0 =A0 =A0=
 at=0A>org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java=
:1326)=0A>=0A>=0A>=A0 =A0 =A0 =A0 at=0A>org.apache.hadoop.hbase.client.HCon=
nectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnec=
tionManager.java:1345)=0A>=A0 =A0 =A0 =A0 at=0A>org.apache.hadoop.hbase.cli=
ent.HTable.incrementColumnValue(HTable.java:870)=0A>=A0 =A0 =A0 =A0 at=0A>o=
rg.apache.hadoop.hbase.client.HTable.incrementColumnValue(HTable.java:850)=
=0A>=A0 =A0 =A0 =A0 <Rest of the stack is inside my code>=0A>=0A>Thanks,=0A=
>Mohamed Ibrahim
---730330825-1954445335-1347854767=:97374--