hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "11 Nov." <nov.eleve...@gmail.com>
Subject OOME when restarting hbase
Date Tue, 12 May 2009 08:46:15 GMT
hi colleagues,


I am doing batup insert on a 20 nodes cluster. Unfortunately the data are
inserted to the last region (sequential insert),

I have already applied the Hbase-1058 patch before the batch insert. But I
got a "big" region with 4450 HStoreFile after

a long run. Now I want to start the hbase, but always got a OOME on
compaction. And the OOME will infect to other regionServer

if one server is OOME.



How to save my data?



My Configuration:

1. hadoop 0.19 branch head

2. hbase 0.19.1 with Hbase-1058 patch

3. 8GRAM (reserved 6GRAM for Hbase in hbase-env.sh)

4. hbase.hstore.compaction.max = 5 (probaly of no use)





2009-05-12 13:55:12,520 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=316, stores=316, storefiles=4450,
storefileIndexSize=251, memcacheSize=0, usedHeap=3941, maxHeap=5333

2009-05-12 13:55:12,520 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: Set stop flag in
regionserver/0:0:0:0:0:0:0:0:62020.compactor

java.lang.OutOfMemoryError: Java heap space

        at java.util.Arrays.copyOf(Arrays.java:2760)

        at java.util.Arrays.copyOf(Arrays.java:2734)

        at java.util.ArrayList.ensureCapacity(ArrayList.java:167)

        at java.util.ArrayList.add(ArrayList.java:351)

        at
org.apache.hadoop.hbase.io.MapFile$Reader.readIndex(MapFile.java:370)

        at
org.apache.hadoop.hbase.io.MapFile$Reader.seekInternal(MapFile.java:462)

        at
org.apache.hadoop.hbase.io.MapFile$Reader.getClosest(MapFile.java:586)

        at
org.apache.hadoop.hbase.io.MapFile$Reader.getClosest(MapFile.java:569)

        at
org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.getClosest(BloomFilterMapFile.java:115)

        at
org.apache.hadoop.hbase.io.HalfMapFileReader.getClosest(HalfMapFileReader.java:152)

        at
org.apache.hadoop.hbase.io.HalfMapFileReader.next(HalfMapFileReader.java:190)

        at
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1133)

        at
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:936)

        at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:727)

        at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:684)

        at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)

2009-05-12 13:55:13,937 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 62020

2009-05-12 13:55:14,003 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 62020: exiting

2009-05-12 13:55:14,503 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer

2009-05-12 13:55:15,627 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 100 on 62020: exiting


    Any suggestions are welcomed! Thanks!

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message