hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Purtell (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HBASE-900) Regionserver memory leak causing OOME during relatively modest bulk importing
Date Sat, 22 Nov 2008 19:58:44 GMT

    [ https://issues.apache.org/jira/browse/HBASE-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12649971#action_12649971
] 

apurtell edited comment on HBASE-900 at 11/22/08 11:57 AM:
-----------------------------------------------------------------

Last night under Heritrix hbase-writer stress I had a regionserver with 2GB heap go down with
an OOME. It was serving 4 regions only. This was with 0.18.1, so the line numbers won't match
up with trunk. 

class [B 27343 2008042041
class [C 11714 966164
class org.apache.hadoop.hbase.HStoreKey 9781 312992
class java.util.TreeMap$Entry 7596 311436
class [Lorg.apache.hadoop.io.WritableComparable; 17 139536

Incidentally the RS was also hosting ROOT so the whole cluster went down. I agree with jgray
this combined with the ROOT SPOF is deadly.

Stack trace of the OOME:

2008-11-22 09:24:40,950 INFO org.apache.hadoop.hbase.regionserver.HRegion: starting compaction
on region content,29308276c599f8a0baca15c224c22ad2,1227337259738
2008-11-22 09:24:56,429 FATAL org.apache.hadoop.hbase.regionserver.HRegionServer: Set stop
flag in regionserver/0:0:0:0:0:0:0:0:60020.compactor
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2786)
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:78)
        at org.apache.hadoop.io.compress.CompressorStream.write(CompressorStream.java:71)
        at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at org.apache.hadoop.hbase.io.ImmutableBytesWritable.write(ImmutableBytesWritable.java:116)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
        at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
        at org.apache.hadoop.io.SequenceFile$RecordCompressWriter.append(SequenceFile.java:1131)
        at org.apache.hadoop.io.SequenceFile$Writer.append(SequenceFile.java:980
)
        at org.apache.hadoop.io.MapFile$Writer.append(MapFile.java:198)
        at org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Writer.append(HStoreFile.java:846)
        at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:988)
        at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:893)
        at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:902)
        at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:860)
        at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:83)

      was (Author: apurtell):
    Last night under Heritrix hbase-writer stress I had a regionserver with 2GB heap go down
with an OOME. It was serving 4 regions only. 

class [B 27343 2008042041
class [C 11714 966164
class org.apache.hadoop.hbase.HStoreKey 9781 312992
class java.util.TreeMap$Entry 7596 311436
class [Lorg.apache.hadoop.io.WritableComparable; 17 139536

Incidentally the RS was also hosting ROOT so the whole cluster went down. I agree with jgray
this combined with the ROOT SPOF is deadly.
  
> Regionserver memory leak causing OOME during relatively modest bulk importing
> -----------------------------------------------------------------------------
>
>                 Key: HBASE-900
>                 URL: https://issues.apache.org/jira/browse/HBASE-900
>             Project: Hadoop HBase
>          Issue Type: Bug
>    Affects Versions: 0.18.1, 0.19.0
>            Reporter: Jonathan Gray
>            Assignee: stack
>            Priority: Blocker
>         Attachments: memoryOn13.png
>
>
> I have recreated this issue several times and it appears to have been introduced in 0.2.
> During an import to a single table, memory usage of individual region servers grows w/o
bounds and when set to the default 1GB it will eventually die with OOME.  This has happened
to me as well as Daniel Ploeg on the mailing list.  In my case, I have 10 RS nodes and OOME
happens w/ 1GB heap at only about 30-35 regions per RS.  In previous versions, I have imported
to several hundred regions per RS with default heap size.
> I am able to get past this by increasing the max heap to 2GB.  However, the appearance
of this in newer versions leads me to believe there is now some kind of memory leak happening
in the region servers during import.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message