hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jim Kellerman (POWERSET)" <Jim.Keller...@microsoft.com>
Subject RE: out of memory error
Date Thu, 16 Oct 2008 16:30:29 GMT
In the future, you will get a more timely response for hbase
questions if you post them on the hbase-user@hadoop.apache.org
mailing list.

In order to address your question, it would be helpful to
know your hardware configuration (memory, # of cores),
any changes you have made to hbase-site.xml, how many
file handles are allocated per process, what else is
running on the same machine as the region server and
what versions of hadoop and hbase you are running.

---
Jim Kellerman, Powerset (Live Search, Microsoft Corporation)

> -----Original Message-----
> From: Rui Xing [mailto:xingrui@gmail.com]
> Sent: Thursday, October 16, 2008 4:52 AM
> To: core-user@hadoop.apache.org
> Subject: out of memory error
>
> Hello List,
>
> We encountered an out-of-memory error in data loading. We have 5 data
> nodes
> and 1 name node distributed on 6 machines. Block-level compression was
> used.
> Following is the log output. Seems the problem was caused in compression.
> Is
> there anybody who ever experienced such error? Any helps or clues are
> appreciated.
>
> 2008-10-15 21:44:33,069 FATAL
> [regionserver/0:0:0:0:0:0:0:0:60020.compactor]
> regionserver.HRegionServer$1(579): Set stop flag in
> regionserver/0:0:0:0:0:0:0:0:60020.compactor
> java.lang.OutOfMemoryError
>         at sun.misc.Unsafe.allocateMemory(Native Method)
>         at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99)
>         at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
>         at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> r.java:108)
>         at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> r.java:115)
>         at
> org.apache.hadoop.io.compress.zlib.ZlibFactory.getZlibDecompressor(ZlibFac
> tory.java:104)
>         at
> org.apache.hadoop.io.compress.DefaultCodec.createDecompressor(DefaultCodec
> .java:80)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(Sequen
> ceFile.java:1458)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1543)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
>         at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292)
>         at
> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<
> init>(HStoreFile.java:635)
>         at
> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.
> <init>(HStoreFile.java:717)
>         at
> org.apache.hadoop.hbase.regionserver.HStoreFile$HalfMapFileReader.<init>(H
> StoreFile.java:915)
>         at
> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:
> 408)
>         at
> org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:263)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.jav
> a:1698)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:481)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:421)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.splitRegion(HRegion.java:815)
>         at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplit
> Thread.java:133)
>         at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitTh
> read.java:86)
> 2008-10-15 21:44:33,661 FATAL
> [regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher]
> regionserver.Flusher(183):
> Replay of hlog required. Forcing server restart
> org.apache.hadoop.hbase.DroppedSnapshotException: region:
> p4p_test,,1224072139042
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.ja
> va:1087)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:985)
>         at
> org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174)
>         at
> org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:91)
> Caused by: java.lang.OutOfMemoryError
>         at sun.misc.Unsafe.allocateMemory(Native Method)
>         at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99)
>         at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
>         at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> r.java:107)
>         at
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> r.java:115)
>         at
> org.apache.hadoop.io.compress.zlib.ZlibFactory.getZlibDecompressor(ZlibFac
> tory.java:104)
>         at
> org.apache.hadoop.io.compress.DefaultCodec.createDecompressor(DefaultCodec
> .java:80)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(Sequen
> ceFile.java:1458)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1555)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
>         at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
>         at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292)
>         at
> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<
> init>(HStoreFile.java:635)
>         at
> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.
> <init>(HStoreFile.java:717)
>         at
> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:
> 413)
>         at
> org.apache.hadoop.hbase.regionserver.HStore.updateReaders(HStore.java:665)
>         at
> org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java
> :640)
>         at
> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:577)
>         at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.ja
> va:1074)
>         ... 3 more
> 2008-10-15 21:44:33,661 INFO
> [regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher]
> regionserver.Flusher(109):
> regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting
> 2008-10-15 21:44:33,665 DEBUG
> [regionserver/0:0:0:0:0:0:0:0:60020.logRoller]
> regionserver.HLog(236): Closing current log writer
> /hbase/log_172.19.139.3_1224070931944_60020/hlog.dat.1224078265898
> 2008-10-15 21:44:33,667 INFO
> [regionserver/0:0:0:0:0:0:0:0:60020.logRoller]
> regionserver.HLog(249): New log writer created at
> /hbase/log_172.19.139.3_1224070931944_60020/hlog.dat.1224078273665
> 2008-10-15 21:44:33,667 INFO
> [regionserver/0:0:0:0:0:0:0:0:60020.logRoller]
> regionserver.LogRoller(93): LogRoller exiting.
> 2008-10-15 21:44:34,910 DEBUG [regionserver/0:0:0:0:0:0:0:0:60020]
> hbase.RegionHistorian(316): Offlined
> 2008-10-15 21:44:34,911 INFO  [regionserver/0:0:0:0:0:0:0:0:60020]
> regionserver.HRegionServer(432): Stopping infoServer
> 2008-10-15 21:44:34,911 INFO  [Acceptor ServerSocket[addr=
> 0.0.0.0/0.0.0.0,port=0,localport=60030]]
> util.ThreadedServer$Acceptor(656):
> Stopping Acceptor
> ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030]
> 2008-10-15 21:44:34,914 INFO  [regionserver/0:0:0:0:0:0:0:0:60020]
> http.SocketListener(212): Stopped SocketListener on 0.0.0.0:60030
>
> Thanks,
> -Ray

Mime
View raw message