Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 50E5F911E for ; Mon, 27 Feb 2012 22:06:40 +0000 (UTC) Received: (qmail 94203 invoked by uid 500); 27 Feb 2012 22:06:38 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 94176 invoked by uid 500); 27 Feb 2012 22:06:38 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 94168 invoked by uid 99); 27 Feb 2012 22:06:38 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Feb 2012 22:06:38 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates 209.85.214.44 as permitted sender) Received: from [209.85.214.44] (HELO mail-bk0-f44.google.com) (209.85.214.44) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 27 Feb 2012 22:06:31 +0000 Received: by bkuw5 with SMTP id w5so1012309bku.31 for ; Mon, 27 Feb 2012 14:05:52 -0800 (PST) Received-SPF: pass (google.com: domain of jbellis@gmail.com designates 10.204.133.210 as permitted sender) client-ip=10.204.133.210; Authentication-Results: mr.google.com; spf=pass (google.com: domain of jbellis@gmail.com designates 10.204.133.210 as permitted sender) smtp.mail=jbellis@gmail.com; dkim=pass header.i=jbellis@gmail.com Received: from mr.google.com ([10.204.133.210]) by 10.204.133.210 with SMTP id g18mr7624071bkt.107.1330380352322 (num_hops = 1); Mon, 27 Feb 2012 14:05:52 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type:content-transfer-encoding; bh=LwIzoeL/qnmSCFq1DwENj+mzT9cgyiNjt/Df+JIa62M=; b=hCfZ8wCo8yNMKsMpsh4hMEFnN6TWuuFiMfLUyzF2I9NhpzgfOlVf8P072al2nb+sHi wocHZ+j0imKhUQJqhuq1uDX0//lFvBpmOP1aDVhqnFghJIy7iMur3eXsbiTUQHecDZFc 61ERi9tLX5T+nOsTFJ8bD6VPZmiSyAwNOGSwc= Received: by 10.204.133.210 with SMTP id g18mr6108283bkt.107.1330380352221; Mon, 27 Feb 2012 14:05:52 -0800 (PST) MIME-Version: 1.0 Received: by 10.204.129.216 with HTTP; Mon, 27 Feb 2012 14:05:32 -0800 (PST) In-Reply-To: <1330118080.31969.YahooMailNeo@web31806.mail.mud.yahoo.com> References: <1330118080.31969.YahooMailNeo@web31806.mail.mud.yahoo.com> From: Jonathan Ellis Date: Mon, 27 Feb 2012 16:05:32 -0600 Message-ID: Subject: Re: Server crashed due to "OutOfMemoryError: Java heap space" To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org What does the heap dump show is using the memory? On Fri, Feb 24, 2012 at 3:14 PM, Feng Qu wrote: > Hello, > > We have a 6-node ring running 0.8.6 on RHEL 6.1. The first node also runs > OpsCenter community. This node has crashed few time recently with > "OutOfMemoryError: Java heap space" while several compactions on few 200-= 300 > GB SSTables were running. We are using 8GB Java heap on host with 96GB RA= M. > > I would appreciate for help to figure out the root cause and solution. > > Feng Qu > > > =A0INFO [GossipTasks:1] 2012-02-22 13:15:59,135 Gossiper.java (line 697) > InetAddress /10.89.74.67 is now dead. > =A0INFO [ScheduledTasks:1] 2012-02-22 13:16:12,114 StatusLogger.java (lin= e 65) > ReadStage =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 0 =A0 =A0 =A0 = =A0 0 =A0 =A0 =A0 =A0 0 > ERROR [CompactionExecutor:10538] 2012-02-22 13:16:12,115 > AbstractCassandraDaemon.java (line 139) Fatal exception in thread > Thread[CompactionExecutor:10538,1, > main] > java.lang.OutOfMemoryError: Java heap space > =A0 =A0 =A0 =A0 at > org.apache.cassandra.io.util.BufferedRandomAccessFile.(BufferedRand= omAccessFile.java:123) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.io.sstable.SSTableScanner.(SSTableScanner.java= :57) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.io.sstable.SSTableReader.getDirectScanner(SSTableRea= der.java:664) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.db.compaction.CompactionIterator.getCollatingIterato= r(CompactionIterator.java:92) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.db.compaction.CompactionIterator.(CompactionIt= erator.java:68) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.db.compaction.CompactionManager.doCompactionWithoutS= izeEstimation(CompactionManager.java:553) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.db.compaction.CompactionManager.doCompaction(Compact= ionManager.java:507) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionMan= ager.java:142) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.db.compaction.CompactionManager$1.call(CompactionMan= ager.java:108) > =A0 =A0 =A0 =A0 at java.util.concurrent.FutureTask$Sync.innerRun(Unknown = Source) > =A0 =A0 =A0 =A0 at java.util.concurrent.FutureTask.run(Unknown Source) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask= (Unknown > Source) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unk= nown > Source) > =A0 =A0 =A0 =A0 at java.lang.Thread.run(Unknown Source) > =A0INFO [GossipTasks:1] 2012-02-22 13:16:12,115 Gossiper.java (line 697) > InetAddress /10.2.128.55 is now dead. > ERROR [Thread-734] 2012-02-22 13:16:48,189 AbstractCassandraDaemon.java > (line 139) Fatal exception in thread Thread[Thread-734,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has s= hut > down > =A0 =A0 =A0 =A0 at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedEx= ecution(DebuggableThreadPoolExecutor.java:60) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.reject(Unknown= Source) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.execute(Unknow= n Source) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.net.MessagingService.receive(MessagingService.java:4= 90) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.= java:136) > ERROR [Thread-68450] 2012-02-22 13:16:48,189 AbstractCassandraDaemon.java > (line 139) Fatal exception in thread Thread[Thread-68450,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has s= hut > down > =A0 =A0 =A0 =A0 at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedEx= ecution(DebuggableThreadPoolExecutor.java:60) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.reject(Unknown= Source) > =A0 =A0 =A0 =A0 at > java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(Unknown > Source) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.execute(Unknow= n Source) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.net.MessagingService.receive(MessagingService.java:4= 90) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.= java:136) > ERROR [Thread-731] 2012-02-22 13:16:48,189 AbstractCassandraDaemon.java > (line 139) Fatal exception in thread Thread[Thread-731,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has s= hut > down > =A0 =A0 =A0 =A0 at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedEx= ecution(DebuggableThreadPoolExecutor.java:60) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.reject(Unknown= Source) > =A0 =A0 =A0 =A0 at > java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(Unknown > Source) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.execute(Unknow= n Source) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.net.MessagingService.receive(MessagingService.java:4= 90) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.= java:136) > ERROR [Thread-736] 2012-02-22 13:16:48,186 AbstractCassandraDaemon.java > (line 139) Fatal exception in thread Thread[Thread-736,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has s= hut > down > =A0 =A0 =A0 =A0 at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedEx= ecution(DebuggableThreadPoolExecutor.java:60) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.reject(Unknown= Source) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.execute(Unknow= n Source) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.net.MessagingService.receive(MessagingService.java:4= 90) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.= java:136) > ERROR [Thread-723] 2012-02-22 13:16:47,746 AbstractCassandraDaemon.java > (line 139) Fatal exception in thread Thread[Thread-723,5,main] > java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has s= hut > down > =A0 =A0 =A0 =A0 at > org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedEx= ecution(DebuggableThreadPoolExecutor.java:60) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.reject(Unknown= Source) > =A0 =A0 =A0 =A0 at > java.util.concurrent.ThreadPoolExecutor.ensureQueuedTaskHandled(Unknown > Source) > =A0 =A0 =A0 =A0 at java.util.concurrent.ThreadPoolExecutor.execute(Unknow= n Source) > =A0 =A0 =A0 =A0 at > org.apache.cassandra.net.MessagingService.receive(MessagingService.java:4= 90) > --=20 Jonathan Ellis Project Chair, Apache Cassandra co-founder of DataStax, the source for professional Cassandra support http://www.datastax.com