hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Xing" <xing...@gmail.com>
Subject Re: out of memory error
Date Fri, 17 Oct 2008 03:44:12 GMT
Hmmmm, I should have sent to hbase mailing list. Please ignore this. Sorry
for the spam.

On Fri, Oct 17, 2008 at 12:30 AM, Jim Kellerman (POWERSET) <
Jim.Kellerman@microsoft.com> wrote:

> In the future, you will get a more timely response for hbase
> questions if you post them on the hbase-user@hadoop.apache.org
> mailing list.
>
> In order to address your question, it would be helpful to
> know your hardware configuration (memory, # of cores),
> any changes you have made to hbase-site.xml, how many
> file handles are allocated per process, what else is
> running on the same machine as the region server and
> what versions of hadoop and hbase you are running.
>
> ---
> Jim Kellerman, Powerset (Live Search, Microsoft Corporation)
>
> > -----Original Message-----
> > From: Rui Xing [mailto:xingrui@gmail.com]
> > Sent: Thursday, October 16, 2008 4:52 AM
> > To: core-user@hadoop.apache.org
> > Subject: out of memory error
> >
> > Hello List,
> >
> > We encountered an out-of-memory error in data loading. We have 5 data
> > nodes
> > and 1 name node distributed on 6 machines. Block-level compression was
> > used.
> > Following is the log output. Seems the problem was caused in compression.
> > Is
> > there anybody who ever experienced such error? Any helps or clues are
> > appreciated.
> >
> > 2008-10-15 21:44:33,069 FATAL
> > [regionserver/0:0:0:0:0:0:0:0:60020.compactor]
> > regionserver.HRegionServer$1(579): Set stop flag in
> > regionserver/0:0:0:0:0:0:0:0:60020.compactor
> > java.lang.OutOfMemoryError
> >         at sun.misc.Unsafe.allocateMemory(Native Method)
> >         at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99)
> >         at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> > r.java:108)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> > r.java:115)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibFactory.getZlibDecompressor(ZlibFac
> > tory.java:104)
> >         at
> >
> org.apache.hadoop.io.compress.DefaultCodec.createDecompressor(DefaultCodec
> > .java:80)
> >         at
> >
> org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(Sequen
> > ceFile.java:1458)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1543)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> >         at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<
> > init>(HStoreFile.java:635)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.
> > <init>(HStoreFile.java:717)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile$HalfMapFileReader.<init>(H
> > StoreFile.java:915)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:
> > 408)
> >         at
> > org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:263)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.jav
> > a:1698)
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:481)
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:421)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.splitRegion(HRegion.java:815)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplit
> > Thread.java:133)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitTh
> > read.java:86)
> > 2008-10-15 21:44:33,661 FATAL
> > [regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher]
> > regionserver.Flusher(183):
> > Replay of hlog required. Forcing server restart
> > org.apache.hadoop.hbase.DroppedSnapshotException: region:
> > p4p_test,,1224072139042
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.ja
> > va:1087)
> >         at
> > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:985)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174)
> >         at
> > org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:91)
> > Caused by: java.lang.OutOfMemoryError
> >         at sun.misc.Unsafe.allocateMemory(Native Method)
> >         at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99)
> >         at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> > r.java:107)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso
> > r.java:115)
> >         at
> >
> org.apache.hadoop.io.compress.zlib.ZlibFactory.getZlibDecompressor(ZlibFac
> > tory.java:104)
> >         at
> >
> org.apache.hadoop.io.compress.DefaultCodec.createDecompressor(DefaultCodec
> > .java:80)
> >         at
> >
> org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(Sequen
> > ceFile.java:1458)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1555)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1442)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431)
> >         at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426)
> >         at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.<
> > init>(HStoreFile.java:635)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader.
> > <init>(HStoreFile.java:717)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java:
> > 413)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStore.updateReaders(HStore.java:665)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java
> > :640)
> >         at
> > org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:577)
> >         at
> >
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.ja
> > va:1074)
> >         ... 3 more
> > 2008-10-15 21:44:33,661 INFO
> > [regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher]
> > regionserver.Flusher(109):
> > regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting
> > 2008-10-15 21:44:33,665 DEBUG
> > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller]
> > regionserver.HLog(236): Closing current log writer
> > /hbase/log_172.19.139.3_1224070931944_60020/hlog.dat.1224078265898
> > 2008-10-15 21:44:33,667 INFO
> > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller]
> > regionserver.HLog(249): New log writer created at
> > /hbase/log_172.19.139.3_1224070931944_60020/hlog.dat.1224078273665
> > 2008-10-15 21:44:33,667 INFO
> > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller]
> > regionserver.LogRoller(93): LogRoller exiting.
> > 2008-10-15 21:44:34,910 DEBUG [regionserver/0:0:0:0:0:0:0:0:60020]
> > hbase.RegionHistorian(316): Offlined
> > 2008-10-15 21:44:34,911 INFO  [regionserver/0:0:0:0:0:0:0:0:60020]
> > regionserver.HRegionServer(432): Stopping infoServer
> > 2008-10-15 21:44:34,911 INFO  [Acceptor ServerSocket[addr=
> > 0.0.0.0/0.0.0.0,port=0,localport=60030]]
> > util.ThreadedServer$Acceptor(656):
> > Stopping Acceptor
> > ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030]
> > 2008-10-15 21:44:34,914 INFO  [regionserver/0:0:0:0:0:0:0:0:60020]
> > http.SocketListener(212): Stopped SocketListener on 0.0.0.0:60030
> >
> > Thanks,
> > -Ray
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message