Return-Path: Delivered-To: apmail-hadoop-hbase-user-archive@locus.apache.org Received: (qmail 31001 invoked from network); 16 Oct 2008 16:31:56 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 16 Oct 2008 16:31:56 -0000 Received: (qmail 32556 invoked by uid 500); 16 Oct 2008 16:31:50 -0000 Delivered-To: apmail-hadoop-hbase-user-archive@hadoop.apache.org Received: (qmail 32533 invoked by uid 500); 16 Oct 2008 16:31:50 -0000 Mailing-List: contact hbase-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hbase-user@hadoop.apache.org Delivered-To: mailing list hbase-user@hadoop.apache.org Received: (qmail 32512 invoked by uid 99); 16 Oct 2008 16:31:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Oct 2008 09:31:50 -0700 X-ASF-Spam-Status: No, hits=-8.0 required=10.0 tests=RCVD_IN_DNSWL_HI,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of Jim.Kellerman@microsoft.com designates 131.107.115.212 as permitted sender) Received: from [131.107.115.212] (HELO smtp.microsoft.com) (131.107.115.212) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 16 Oct 2008 16:30:41 +0000 Received: from tk5-exhub-c104.redmond.corp.microsoft.com (157.54.88.97) by TK5-EXGWY-E801.partners.extranet.microsoft.com (10.251.56.50) with Microsoft SMTP Server (TLS) id 8.1.291.1; Thu, 16 Oct 2008 09:31:02 -0700 Received: from NA-EXMSG-C103.redmond.corp.microsoft.com ([157.54.110.53]) by tk5-exhub-c104.redmond.corp.microsoft.com ([157.54.88.97]) with mapi; Thu, 16 Oct 2008 09:30:57 -0700 From: "Jim Kellerman (POWERSET)" To: "core-user@hadoop.apache.org" , "hbase-user@hadoop.apache.org" Date: Thu, 16 Oct 2008 09:30:29 -0700 Subject: RE: out of memory error Thread-Topic: out of memory error Thread-Index: AckvpR1/8D+5WuSZTHe1vab0JqiZcgABrrZw Message-ID: References: <568550cd0810160451v6177b88el6fcd841ec96f345c@mail.gmail.com> In-Reply-To: <568550cd0810160451v6177b88el6fcd841ec96f345c@mail.gmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Virus-Checked: Checked by ClamAV on apache.org In the future, you will get a more timely response for hbase questions if you post them on the hbase-user@hadoop.apache.org mailing list. In order to address your question, it would be helpful to know your hardware configuration (memory, # of cores), any changes you have made to hbase-site.xml, how many file handles are allocated per process, what else is running on the same machine as the region server and what versions of hadoop and hbase you are running. --- Jim Kellerman, Powerset (Live Search, Microsoft Corporation) > -----Original Message----- > From: Rui Xing [mailto:xingrui@gmail.com] > Sent: Thursday, October 16, 2008 4:52 AM > To: core-user@hadoop.apache.org > Subject: out of memory error > > Hello List, > > We encountered an out-of-memory error in data loading. We have 5 data > nodes > and 1 name node distributed on 6 machines. Block-level compression was > used. > Following is the log output. Seems the problem was caused in compression. > Is > there anybody who ever experienced such error? Any helps or clues are > appreciated. > > 2008-10-15 21:44:33,069 FATAL > [regionserver/0:0:0:0:0:0:0:0:60020.compactor] > regionserver.HRegionServer$1(579): Set stop flag in > regionserver/0:0:0:0:0:0:0:0:60020.compactor > java.lang.OutOfMemoryError > at sun.misc.Unsafe.allocateMemory(Native Method) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:99) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) > at > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.(ZlibDecompress= o > r.java:108) > at > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.(ZlibDecompress= o > r.java:115) > at > org.apache.hadoop.io.compress.zlib.ZlibFactory.getZlibDecompressor(ZlibFa= c > tory.java:104) > at > org.apache.hadoop.io.compress.DefaultCodec.createDecompressor(DefaultCode= c > .java:80) > at > org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(Seque= n > ceFile.java:1458) > at > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1543) > at > org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1442) > at > org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1431) > at > org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1426) > at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292) > at > org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.= < > init>(HStoreFile.java:635) > at > org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader= . > (HStoreFile.java:717) > at > org.apache.hadoop.hbase.regionserver.HStoreFile$HalfMapFileReader.(= H > StoreFile.java:915) > at > org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java= : > 408) > at > org.apache.hadoop.hbase.regionserver.HStore.(HStore.java:263) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.ja= v > a:1698) > at > org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:481) > at > org.apache.hadoop.hbase.regionserver.HRegion.(HRegion.java:421) > at > org.apache.hadoop.hbase.regionserver.HRegion.splitRegion(HRegion.java:815= ) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSpli= t > Thread.java:133) > at > org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitT= h > read.java:86) > 2008-10-15 21:44:33,661 FATAL > [regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher] > regionserver.Flusher(183): > Replay of hlog required. Forcing server restart > org.apache.hadoop.hbase.DroppedSnapshotException: region: > p4p_test,,1224072139042 > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.j= a > va:1087) > at > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:985) > at > org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174= ) > at > org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:91) > Caused by: java.lang.OutOfMemoryError > at sun.misc.Unsafe.allocateMemory(Native Method) > at java.nio.DirectByteBuffer.(DirectByteBuffer.java:99) > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) > at > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.(ZlibDecompress= o > r.java:107) > at > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.(ZlibDecompress= o > r.java:115) > at > org.apache.hadoop.io.compress.zlib.ZlibFactory.getZlibDecompressor(ZlibFa= c > tory.java:104) > at > org.apache.hadoop.io.compress.DefaultCodec.createDecompressor(DefaultCode= c > .java:80) > at > org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(Seque= n > ceFile.java:1458) > at > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1555) > at > org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1442) > at > org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1431) > at > org.apache.hadoop.io.SequenceFile$Reader.(SequenceFile.java:1426) > at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292) > at > org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.= < > init>(HStoreFile.java:635) > at > org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader= . > (HStoreFile.java:717) > at > org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java= : > 413) > at > org.apache.hadoop.hbase.regionserver.HStore.updateReaders(HStore.java:665= ) > at > org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.jav= a > :640) > at > org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:577) > at > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.j= a > va:1074) > ... 3 more > 2008-10-15 21:44:33,661 INFO > [regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher] > regionserver.Flusher(109): > regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting > 2008-10-15 21:44:33,665 DEBUG > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] > regionserver.HLog(236): Closing current log writer > /hbase/log_172.19.139.3_1224070931944_60020/hlog.dat.1224078265898 > 2008-10-15 21:44:33,667 INFO > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] > regionserver.HLog(249): New log writer created at > /hbase/log_172.19.139.3_1224070931944_60020/hlog.dat.1224078273665 > 2008-10-15 21:44:33,667 INFO > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] > regionserver.LogRoller(93): LogRoller exiting. > 2008-10-15 21:44:34,910 DEBUG [regionserver/0:0:0:0:0:0:0:0:60020] > hbase.RegionHistorian(316): Offlined > 2008-10-15 21:44:34,911 INFO [regionserver/0:0:0:0:0:0:0:0:60020] > regionserver.HRegionServer(432): Stopping infoServer > 2008-10-15 21:44:34,911 INFO [Acceptor ServerSocket[addr=3D > 0.0.0.0/0.0.0.0,port=3D0,localport=3D60030]] > util.ThreadedServer$Acceptor(656): > Stopping Acceptor > ServerSocket[addr=3D0.0.0.0/0.0.0.0,port=3D0,localport=3D60030] > 2008-10-15 21:44:34,914 INFO [regionserver/0:0:0:0:0:0:0:0:60020] > http.SocketListener(212): Stopped SocketListener on 0.0.0.0:60030 > > Thanks, > -Ray