hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-868) Incrementing binary rows cause strange behavior once table splits
Date Sun, 07 Sep 2008 23:40:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629020#action_12629020
] 

stack commented on HBASE-868:
-----------------------------

Reviewed the patch.

I added javadoc here:

{code}
+  // WritableComparator that takes account of meta keys.
+  public static class HStoreKeyWritableComparator extends WritableComparator {
+    private final HRegionInfo hri;
{code}

... and removed the creation of new objects done in the below:

{code}
Index: src/java/org/apache/hadoop/hbase/master/MetaRegion.java
===================================================================
--- src/java/org/apache/hadoop/hbase/master/MetaRegion.java (revision 692702)
+++ src/java/org/apache/hadoop/hbase/master/MetaRegion.java (working copy)
@@ -20,7 +20,10 @@
 package org.apache.hadoop.hbase.master;

 import org.apache.hadoop.hbase.HConstants;
+import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.HTableDescriptor;
 import org.apache.hadoop.hbase.HServerAddress;
+import org.apache.hadoop.hbase.HStoreKey;
 import org.apache.hadoop.hbase.util.Bytes;


@@ -90,7 +93,9 @@
   public int compareTo(MetaRegion other) {
     int result = Bytes.compareTo(this.regionName, other.getRegionName());
     if(result == 0) {
-      result = Bytes.compareTo(this.startKey, other.getStartKey());
+      HRegionInfo hri = new HRegionInfo(HTableDescriptor.META_TABLEDESC, 
+          HConstants.EMPTY_END_ROW, HConstants.EMPTY_END_ROW);
+      result = HStoreKey.compareTwoRowKeys(hri, this.startKey, other.getStartKey());
       if (result == 0) {
         // Might be on different host?
         result = this.server.compareTo(other.server);
Index: src/java/org/apache/hadoop/hbase/client/MetaScanner.java
===================================================================
--- src/java/org/apache/hadoop/hbase/client/MetaScanner.java    (revision 692702)
+++ src/java/org/apache/hadoop/hbase/client/MetaScanner.java    (working copy)
@@ -5,6 +5,7 @@
 import org.apache.hadoop.hbase.HBaseConfiguration;
 import org.apache.hadoop.hbase.HConstants;
 import org.apache.hadoop.hbase.HRegionInfo;
+import org.apache.hadoop.hbase.HStoreKey;
 import org.apache.hadoop.hbase.io.RowResult;
 import org.apache.hadoop.hbase.util.Bytes;

@@ -47,6 +48,7 @@
           HRegionInfo.createRegionName(tableName, null, ZEROES);

     // Scan over each meta region
+    HRegionInfo hri;
     do {
       ScannerCallable callable = new ScannerCallable(connection,
         META_TABLE_NAME, COLUMN_FAMILY_ARRAY, startRow, LATEST_TIMESTAMP, null);
@@ -62,12 +64,13 @@
         } while(visitor.processRow(r));
         // Advance the startRow to the end key of the current region
         startRow = callable.getHRegionInfo().getEndKey();
+        hri = new HRegionInfo(callable.getHRegionInfo());
       } finally {
         // Close scanner
         callable.setClose();
         connection.getRegionServerWithRetries(callable);
       }
-    } while (Bytes.compareTo(startRow, LAST_ROW) != 0);
+    } while (HStoreKey.compareTwoRowKeys(hri, startRow, LAST_ROW) != 0);
{code}

Otherwise, the patch looks fine.

This kinda change looks unnecessary in SFS but I'm afraid to change it since the patch works:
{code}
-      keys[i] = new HStoreKey();
+      keys[i] = new HStoreKey(HConstants.EMPTY_BYTE_ARRAY, this.store.getHRegionInfo());
{code}

I'm running unit tests now.  I'll then run the amended PE test to ensure binary keys work.

> Incrementing binary rows cause strange behavior once table splits
> -----------------------------------------------------------------
>
>                 Key: HBASE-868
>                 URL: https://issues.apache.org/jira/browse/HBASE-868
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.2.1
>         Environment: 4 nodes: 1 master/namenode, 3 regionserver/datanodes
>            Reporter: Jonathan Gray
>            Assignee: Jonathan Gray
>            Priority: Blocker
>             Fix For: 0.2.1
>
>         Attachments: hbase-868-0.2-v2.patch, hbase-868-0.2-v3.patch, hbase-868-0.2-v4.patch,
hbase-868-0.2-v5.patch, hbase-868-branch-0.2.patch, hbase-hbase-regionserver-web3.streamy.com.log,
master_regionsinsources.png, regionserver_regionlist.png
>
>
> We're now using incrementing binary row keys and to this point had only been doing small
tests with them, never having actually had a table split.
> I'm still working through the logs but it seems that there's a problem somewhere with
startKey and endKeys.
> Binary in general is not well supported (inconsistent in display in the logs, very odd
rendering in the web ui, hard to interpret in the shell, etc..)  Once we figure out these
serious bugs we will spend some time trying to clean that up.  But right now this makes things
even harder to debug.
> The actual symptoms are that my import eventually started to throw (in the client and
on the region server):
> org.apache.hadoop.hbase.regionserver.WrongRegionException: org.apache.hadoop.hbase.regionserver.WrongRegionException:
Requested row out of range for HRegion sources,,1220546297947, startKey='', getEndKey()='
> �', row='c�'
>         at org.apache.hadoop.hbase.regionserver.HRegion.checkRow(HRegion.java:1775)
>         at org.apache.hadoop.hbase.regionserver.HRegion.obtainRowLock(HRegion.java:1831)
>         at org.apache.hadoop.hbase.regionserver.HRegion.batchUpdate(HRegion.java:1387)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdate(HRegionServer.java:1145)
>         at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:616)
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:473)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> There are 3 regionservers, but this error only happens on one of them (while the other
two always continue normally, allowing updates to this same table).
> The regionserver that this happens on is special for two reasons, one it is hosting the
META table.  And secondly it also hosts the first region in the table, startkey = ''.  I'm
unsure which is the cause, I have a clue leading to both.
> After about 15 minutes, the regionserver sees:
> 2008-09-04 09:52:57,948 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Started memcache
flush for region .META.,,1. Current region memcache size 24.5k
> 2008-09-04 09:52:58,003 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/.META./1028785192/historian/mapfiles/8699673838203663799
with 106 entries, sequence id 25341510, data size 8.9k, file size 10.6k
> 2008-09-04 09:52:58,050 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Added /hbase/.META./1028785192/info/mapfiles/1791564557665476834
with 96 entries, sequence id 25341510, data size 14.2k, file size 15.8k
> 2008-09-04 09:52:58,050 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished
memcache flush for region .META.,,1 in 102ms, sequence id=25341510, compaction requested=true
> 2008-09-04 09:52:58,050 DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread:
Compaction requested for region: .META.,,1
> 2008-09-04 09:52:58,051 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction
on region .META.,,1
> 2008-09-04 09:52:58,055 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Compaction
size of 1028785192/historian: 41.9k; Skipped 1 files , size: 21957
> 2008-09-04 09:52:58,088 DEBUG org.apache.hadoop.hbase.regionserver.HStore: started compaction
of 2 files into /hbase/.META./compaction.dir/1028785192/historian/mapfiles/6948796056606699674
> 2008-09-04 09:52:58,128 DEBUG org.apache.hadoop.hbase.regionserver.HStore: moving /hbase/.META./compaction.dir/1028785192/historian/mapfiles/6948796056606699674
to /hbase/.META./1028785192/historian/mapfiles/75733875840914142
> 2008-09-04 09:52:58,175 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Completed
compaction of 1028785192/historian store size is 41.1k; time since last major compaction:
5426 seconds
> 2008-09-04 09:52:58,179 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Compaction
size of 1028785192/info: 61.9k; Skipped 0 files , size: 0
> 2008-09-04 09:52:58,192 DEBUG org.apache.hadoop.hbase.regionserver.HStore: started compaction
of 3 files into /hbase/.META./compaction.dir/1028785192/info/mapfiles/7781013568996125923
> 2008-09-04 09:52:58,260 DEBUG org.apache.hadoop.hbase.regionserver.HStore: moving /hbase/.META./compaction.dir/1028785192/info/mapfiles/7781013568996125923
to /hbase/.META./1028785192/info/mapfiles/2187291308709057119
> 2008-09-04 09:52:58,290 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Completed
compaction of 1028785192/info store size is 61.0k; time since last major compaction: 32534
seconds
> 2008-09-04 09:52:58,296 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction
completed on region .META.,,1 in 0sec
> 2008-09-04 09:53:09,620 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner
-2085474968086468199 lease expired
> 2008-09-04 09:54:35,449 INFO org.apache.hadoop.hbase.regionserver.LogRoller: Rolling
hlog. Number of entries: 30009
> Following this, insertion continues normally.  This leads me to believe there's an issue
with the META table memcache, but oddly the other regions of this table on other regionservers
continue on fine.
> As for the hosting the first region of the table on this region server, it seems to be
consistent that I get the row out of range errors when looking for a region with startkey
= '', although there are 5 other regions on this RS.
> Will attach full logs from master and three RS.  Also a couple screenshots showing weird
behavior in listing the regions in the table.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message