hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HBASE-1415) Stuck on memcache flush
Date Wed, 13 May 2009 07:08:45 GMT

    [ https://issues.apache.org/jira/browse/HBASE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708802#action_12708802
] 

Andrew Purtell commented on HBASE-1415:
---------------------------------------

This is 1058 not working as intended. Too many store files, compaction is requested, it completes,
but does not actually compact anything. So, the gate never comes up.

> Stuck on memcache flush
> -----------------------
>
>                 Key: HBASE-1415
>                 URL: https://issues.apache.org/jira/browse/HBASE-1415
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.20.0
>
>
> Latest trunk, regionserver gets stuck doing below:
> {code}
> 2009-05-13 03:05:28,126 [regionserver/0:0:0:0:0:0:0:0:60021.logRoller] INFO org.apache.hadoop.hbase.regionserver.HLog:
Closed hdfs://aa0-000-12.u.powerset.com:9000/hbasetrunk2/.logs/aa0-000-13.u.powerset.com_1242178588540_60021/hlog.dat.1242183899413,
entries=10051
> 2009-05-13 03:05:28,126 [regionserver/0:0:0:0:0:0:0:0:60021.logRoller] DEBUG org.apache.hadoop.hbase.regionserver.HLog:
Found 0 logs to remove  out of total 53; oldest outstanding seqnum is 6838308 from region
TestTable,0842399686,1242180442452
> 2009-05-13 03:05:28,737 [IPC Server handler 9 on 60021] INFO org.apache.hadoop.hbase.regionserver.MemcacheFlusher:
Forced flushing of TestTable,0107641622,1242181965182 because global memcache limit of 497.8m
exceeded; currently 513.3m and flushing till 311.1m
> 2009-05-13 03:05:28,737 [IPC Server handler 9 on 60021] DEBUG org.apache.hadoop.hbase.regionserver.MemcacheFlusher:
Too many store files in store info: 5, waiting
> 2009-05-13 03:05:28,737 [IPC Server handler 9 on 60021] DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread:
Compaction requested for region TestTable,0107641622,1242181965182/614435386 because: regionserver/0:0:0:0:0:0:0:0:60021.cacheFlusher
> 2009-05-13 03:06:07,143 [regionserver/0:0:0:0:0:0:0:0:60021.compactor] DEBUG org.apache.hadoop.hbase.regionserver.Store:
Completed compaction of info; store size is 168.0m
> 2009-05-13 03:06:07,143 [regionserver/0:0:0:0:0:0:0:0:60021.compactor] INFO org.apache.hadoop.hbase.regionserver.HRegion:
compaction completed on region TestTable,0092188538,1242183375224 in 3mins, 58sec
> 2009-05-13 03:06:58,757 [IPC Server handler 9 on 60021] DEBUG org.apache.hadoop.hbase.regionserver.MemcacheFlusher:
Too many store files in store info: 5, waiting
> 2009-05-13 03:06:58,757 [IPC Server handler 9 on 60021] DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread:
Compaction requested for region TestTable,0107641622,1242181965182/614435386 because: regionserver/0:0:0:0:0:0:0:0:60021.cacheFlusher
> 2009-05-13 03:08:28,767 [IPC Server handler 9 on 60021] DEBUG org.apache.hadoop.hbase.regionserver.MemcacheFlusher:
Too many store files in store info: 5, waiting
> 2009-05-13 03:08:28,767 [IPC Server handler 9 on 60021] DEBUG org.apache.hadoop.hbase.regionserver.CompactSplitThread:
Compaction requested for region TestTable,0107641622,1242181965182/614435386 because: regionserver/0:0:0:0:0:0:0:0:60021.cacheFlusher
> 2009-05-13 03:09:58,777 [IPC Server handler 9 on 60021] DEBUG org.apache.hadoop.hbase.regionserver.MemcacheFlusher:
Too many store files in store info: 5, waiting
> ...
> {code}
> Thread dumping I see a few threads blocked here:
> {code}
> "IPC Server handler 4 on 60021" daemon prio=10 tid=0x00007f42982ee000 nid=0x599b waiting
for monitor entry [0x000000004380b000..0x000000004380bb00]
>    java.lang.Thread.State: BLOCKED (on object monitor)
>         at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:294)
>         - waiting to lock <0x00007f42aac8cbd8> (a org.apache.hadoop.hbase.regionserver.MemcacheFlusher)
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1800)
>         at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:642)
>         at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:911)
> ..
> {code}
> And this is a good while after my MR upload failed.
> Assigning myself since I can reproduce easy enough.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message