Mailing-List: contact user-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hbase.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type;
  b=NL1xOi4/P5twOZuPwSfUczycfNecBAybSqvXi4sCZRlTzV20ibasOkVXSVzAjK1k0ly33bx3v3O87gA4GqtMSEVhd8VjtmXwkq1i6Fw+ZmALR85/dcPPVe/+HPybDGykjUBqaK/5pDK+F7lZ67app6WmiKHJr4ftVROmsgsnwkw=;
Message-ID: <584545.20008.qm@web52304.mail.re2.yahoo.com>
Date: Fri, 6 Aug 2010 14:59:57 -0700 (PDT)
From: Stuart Smith <stu24mail@yahoo.com>
Subject: Batch puts interrupted ... Requested row out of range for HRegion
 filestore ...org.apache.hadoop.hbase.client.RetriesExhaustedException:
To: user@hbase.apache.org
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Hello,

  I'm running hbase 0.20.5, and seeing Puts() fail repeatedly when trying to insert a specific item into the database.

Client side I see:

org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server Some server, retryOnlyOne=true, index=0, islastrow=true, tries=9, numtries=10, i=0, listsize=1, region=filestore,bdfa9f2173033330cfae81ece08f75f0002bf3f3a54cde6bbf9192f0187e275b,1279604506836 for region filestore,

I then looked up which node was hosting the given region (filestore,bdfa9f2173033330cfae81ece08f75f0002bf3f3a54cde6bbf9192f0187e275b) on the gui, found the following debug message in the regionserver log:

2010-08-06 14:23:47,414 DEBUG org.apache.hadoop.hbase.regionserver.HRegionServer: Batch puts interrupted at index=0 because:Requested row out of range for HRegion filestore,bdfa9f2173033330cfae81ece08f75f0002bf3f3a54cde6bbf9192f0187e275b,1279604506836, startKey='bdfa9f2173033330cfae81ece08f75f0002bf3f3a54cde6bbf9192f0187e275b', getEndKey()='be0bc7b3f8bc2a30910b9c758b47cdb730a4691e93f92abb857a2dcc7aefa633', row='be1681910b02db5da061659c2cb08f501a135c2f065559a37a1761bf6e203d1d'


Which appears to be coming from:

/regionserver/HRegionServer.java:1786:      LOG.debug("Batch puts interrupted at index=" + i + " because:" +

Which is coming from:

./java/org/apache/hadoop/hbase/regionserver/HRegion.java:1658:      throw new WrongRegionException("Requested row out of range for " +

This happens repeatedly on a specific item over at least a day or so, even when not much is happening with the cluster.

As far as I can tell, it looks like the logic to select the correct region for a given row is wrong. The row is indeed not in the correct range (at least from what I can tell of the exception thrown), and the check in HRegion.java:1658:

  /** Make sure this is a valid row for the HRegion */
  private void checkRow(final byte [] row) throws IOException {
    if(!rowIsInRange(regionInfo, row)) {

Is correctly rejecting the Put().

So it appears the error would be somewhere in: 
HRegion.java:1550: 
  private void put(final Map<byte [],List<KeyValue>> familyMap,
      boolean writeToWAL) throws IOException {

Which appears to be the actual guts of the insert operation.
However, I don't know enough about the design of HRegions to really decipher this method. I'll dig into it more, but I thought it might be more efficient just to ask you guys first.

Any ideas? 

I can update to 0.20.6, but I don't see any fixed jira's on 0.20.6 that seem related.. I could be wrong. I'm not sure what I should do next. Any more information you guys need?

Note that I am inserting file into the database, and using it's sha256sum as the key. And the file that is failing does indeed have a sha that corresponds to the key in the message above (and is out of range).

Take care,
  -stu