hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jean-Daniel Cryans (JIRA)" <j...@apache.org>
Subject [jira] Created: (HBASE-2018) Updates to .META. blocked under high MemStore load
Date Mon, 30 Nov 2009 22:33:20 GMT
Updates to .META. blocked under high MemStore load

                 Key: HBASE-2018
                 URL: https://issues.apache.org/jira/browse/HBASE-2018
             Project: Hadoop HBase
          Issue Type: Bug
    Affects Versions: 0.20.2
            Reporter: Jean-Daniel Cryans
            Assignee: Jean-Daniel Cryans
            Priority: Blocker
             Fix For: 0.20.3, 0.21.0

I discovered this on Lars' cluster. The symptom was the good old:

09/11/30 08:10:26 INFO mapred.JobClient: Task Id : attempt_200911250121_0011_r_000010_1, Status
org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact region server
Some server, retryOnlyOne=true, index=0, islastrow=false, tries=9, numtries=10, i=14, listsize=20,
region=prev-docs,de68fb97795ef3d936a3f10ff8790253,1259573366564 for region prev-docs,ccea967e66ccb53d83c48849c3a23f21,1259542138868,
row 'ccff8cd4ca871c41f4fa7d44cffed962', but failed after 10 attempts.
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1120)
        at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1201)
        at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:605)
        at org.apache.hadoop.hbase.client.HTable.put(HTable.java:470)
        at org.apache.hadoop.hbase.mapreduce.TableOutputFormat$TableRecordW

But the load wasn't that heavy, just lots of splitting going on. Looking at the logs, I see
a split taking more than 4 minutes which is explained by this happening on the RS hosting
.META. :

2009-11-30 08:08:39,922 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Forced
flushing of prev-docs,2c9d51e57b20decd5c6419d23ede822b,1259542273901 because global memstore
limit of 1.6g exceeded; currently 1.6g and flushing till 1021.9m
2009-11-30 08:12:33,743 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore
flush of ~22.9m for region prev-docs,c8fea4fbbc41e746d960854ed4d41dd6,1259587143838 in 14160ms,
sequence id=13677, compaction requested=false
2009-11-30 08:12:33,744 INFO org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Forced
flushing of prev-docs,39c2995d955c041d21f4dc4a0d0dbf6c,1259587061295 because global memstore
limit of 1.6g exceeded; currently 1.0g and flushing till 1021.9m

So we should not block updates to .META. for any reason. I'm pretty sure this issue explains
other issues we've seen on the mailing list.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message