hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (HBASE-1006) Resevoir of memory not freed on OOME
Date Tue, 25 Nov 2008 06:35:44 GMT

     [ https://issues.apache.org/jira/browse/HBASE-1006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack resolved HBASE-1006.
--------------------------

    Resolution: Won't Fix

Resevoir is cleared in call to abort.  It happens silently. With above exception we do not
abort because in BatchUpdate, we see the DSE as an IOE as it goes by, not as an OOME.  OOMEs
can happen anywhere.  Thats a bit of a problem given as there is no single entry point into
HRS.

Resolving as fixed as part of HBASE-1020.  Over there, I'll add a check for OOME everywhere
in HRS.  If an OOME, will abort.   Abort will then do the clear of the resevoir so the abort
can run smoothly.  Will also dump metrics.

As to whether abort and shutdown on OOME is the right thing to do, thats we should discuss.

> Resevoir of memory not freed on OOME
> ------------------------------------
>
>                 Key: HBASE-1006
>                 URL: https://issues.apache.org/jira/browse/HBASE-1006
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>
> Here is the OOME:
> {code}
> #
> 2008-11-17 17:04:18,812 FATAL org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Replay
of hlog required. Forcing server shutdown
> #
> org.apache.hadoop.hbase.DroppedSnapshotException: region: streamitems,^@^@^@^@^A�?�,1226968617756
> #
>         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:865)
> #
>         at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:761)
> #
>         at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushRegion(MemcacheFlusher.java:179)
> #
>         at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.flushSomeRegions(MemcacheFlusher.java:232)
> #
>         at org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:213)
> #
>         at org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1312)
> #
>         at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
> #
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> #
>         at java.lang.reflect.Method.invoke(Method.java:597)
> #
>         at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:634)
> #
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:888)
> #
> Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded
> #
>         at org.apache.hadoop.hbase.util.Bytes.readByteArray(Bytes.java:62)
> #
>         at org.apache.hadoop.hbase.HStoreKey.readFields(HStoreKey.java:589)
> #
>         at org.apache.hadoop.io.MapFile$Writer.checkKey(MapFile.java:213)
> #
>         at org.apache.hadoop.io.MapFile$Writer.append(MapFile.java:192)
> #
>         at org.apache.hadoop.hbase.io.BloomFilterMapFile$Writer.append(BloomFilterMapFile.java:201)
> #
>         at org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java:674)
> #
>         at org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:627)
> #
>         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:852)
> #
>         at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:852)
> {code}
> Looking in heap, the resevoir had not been released.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message