Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (athena.apache.org: domain of m0brhm@gmail.com designates
 209.85.216.169 as permitted sender)
MIME-Version: 1.0
Sender: m0brhm@gmail.com
In-Reply-To: <1347838400.96816.YahooMailNeo@web121706.mail.ne1.yahoo.com>
References: 
 <CAB7cnggHneqBWK8KqRntA2nZuUvJrzVesNuKjWAtmKCE2LefzA@mail.gmail.com>
	<1347838400.96816.YahooMailNeo@web121706.mail.ne1.yahoo.com>
Date: Sun, 16 Sep 2012 22:13:19 -0400
Message-ID: 
 <CAB7cnggXZdUr1p4yvxY=D4S=G1o-BDqgYOuiOkTsz1RASQGpiw@mail.gmail.com>
Subject: Re: java.io.IOEcxeption key k1 followed by a smaller key k2
From: Mohamed Ibrahim <mibrahim@mibrahim.net>
To: user@hbase.apache.org, lars hofhansl <lhofhansl@yahoo.com>,
 Stack <stack@duboce.net>
Content-Type: multipart/alternative; boundary=20cf303b39f9371f0904c9dc513b

--20cf303b39f9371f0904c9dc513b
Content-Type: text/plain; charset=ISO-8859-1

Hello Lars / Stack,

Thank you for responding.

The date on the files is March 9th 2012. It's been up since then, I
restarted hbase and hadoop once. I only have a single node that I'm running
my tests on. I'm currently running 0.92.1 on hadoop 1.0 . I hope I'm using
the correct mix.

I'm not using any external tools other than hbase java api, and inspecting
the data using the shell. I ran my program, the following day I found the
stack dump on the console. I checked by scanning the table that had the
exception for the smaller key k1, with limit 2 from the shell and no
exceptions were thrown. I can also see that the following key is larger
than the first one, so nothing is wrong.

I faced the same exception before and it happened more frequently when I
did a lot of Puts and Deletes using Htable.batch . Essentially I was
updating several inverted indexes each in its own table on the data of rows
when they get inserted, and instead of doing single Deletes & Puts on the
indexes I used batch. batch improved the performance but threw this
exception more. I stopped using batch and now doing single Puts & Deletes.
The exception rarely gets thrown, but still sporadically gets thrown.

I will read about the hfile tool, thanks for the pointers. I will also try
figure out a set of steps that would repeat the exception so it is more
helpful. I will also try 0.94.1 with batch and see if it will happen again,
and will let you know and will file a bug if I can repeat it consistently.

Thank you,
Mohamed Ibrahim

On Sun, Sep 16, 2012 at 7:33 PM, lars hofhansl <lhofhansl@yahoo.com> wrote:

> Hmm... HBASE-6579 gets rid of that check, because we thought it no longer
> necessary.
> No do you remember what you did leading up to this?
> Did you write these HFiles with some other tool? Done some bulk import,
> etc?
>
>
> -- Lars
>
>
>
> ________________________________
>  From: Mohamed Ibrahim <m0brhm@gmail.com>
> To: user@hbase.apache.org
> Sent: Sunday, September 16, 2012 5:59 AM
> Subject: java.io.IOEcxeption key k1 followed by a smaller key k2
>
> Hello All,
>
> I am using hbase 0.92.1 on hadoop 1 . I am getting those exceptions, and it
> seems to me that it means that the hbase file is not sorted in order. So
> when the scanner goes through it, it finds a smaller key after its current.
>
> Is that related to https://issues.apache.org/jira/browse/HBASE-6579 ??
>
> It looks like upgrading to 0.94.1 (current stable) won't fix the issue. Any
> recommendations ??
>
> Here is the stack dump:
>         at
>
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:266)
>         at
>
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:138)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:2945)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:2901)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:2918)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.getLastIncrement(HRegion.java:3660)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegion.incrementColumnValue(HRegion.java:3844)
>         at
>
> org.apache.hadoop.hbase.regionserver.HRegionServer.incrementColumnValue(HRegionServer.java:2982)
>         at sun.reflect.GeneratedMethodAccessor25.invoke(Unknown Source)
>         at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at
>
> org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:364)
>         at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1326)
>
>
>         at
>
> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getRegionServerWithRetries(HConnectionManager.java:1345)
>         at
> org.apache.hadoop.hbase.client.HTable.incrementColumnValue(HTable.java:870)
>         at
> org.apache.hadoop.hbase.client.HTable.incrementColumnValue(HTable.java:850)
>         <Rest of the stack is inside my code>
>
> Thanks,
> Mohamed Ibrahim
>

--20cf303b39f9371f0904c9dc513b--