Return-Path: Delivered-To: apmail-cassandra-dev-archive@www.apache.org Received: (qmail 73025 invoked from network); 18 Sep 2010 18:47:36 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 18 Sep 2010 18:47:36 -0000 Received: (qmail 38703 invoked by uid 500); 18 Sep 2010 18:47:36 -0000 Delivered-To: apmail-cassandra-dev-archive@cassandra.apache.org Received: (qmail 38644 invoked by uid 500); 18 Sep 2010 18:47:35 -0000 Mailing-List: contact dev-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list dev@cassandra.apache.org Received: (qmail 38636 invoked by uid 500); 18 Sep 2010 18:47:35 -0000 Delivered-To: apmail-incubator-cassandra-dev@incubator.apache.org Received: (qmail 38633 invoked by uid 99); 18 Sep 2010 18:47:35 -0000 Received: from Unknown (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Sep 2010 18:47:35 +0000 X-ASF-Spam-Status: No, hits=0.7 required=10.0 tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.216.175] (HELO mail-qy0-f175.google.com) (209.85.216.175) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 18 Sep 2010 18:47:13 +0000 Received: by qyk31 with SMTP id 31so1904667qyk.6 for ; Sat, 18 Sep 2010 11:46:52 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.217.149 with SMTP id hm21mr4704605qcb.121.1284835611246; Sat, 18 Sep 2010 11:46:51 -0700 (PDT) Received: by 10.229.79.194 with HTTP; Sat, 18 Sep 2010 11:46:51 -0700 (PDT) X-Originating-IP: [166.132.177.0] In-Reply-To: References: Date: Sat, 18 Sep 2010 13:46:51 -0500 Message-ID: Subject: Re: OOM when running hector's unit tests at 0.7.0 From: Nate McCall To: hector-dev@googlegroups.com Cc: cassandra-dev@incubator.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org Ran, have you tried to reproduce this using cassandra's python system tests? Those have coverage of the system methods as well. On 9/18/10, Nate McCall wrote: > An additional point - we just added the system_* methods and this test > class does an add, rename then drop for both CF and keyspace. > > On 9/18/10, Ran Tavory wrote: >> I started seeing OOM when running hector's unit-tests on 0.7.0. The OOM >> is >> in cassandra's code so either there had been recent changes within >> cassandra >> or it's just that we added new coverage which causes this OOM. >> >> On my mac it's 100% repro but other hector devs have reported they don't >> get >> this OOM. Moreover, it I place a Thread.sleep(5000) in one of the tests >> the >> OOM goes away (but other errors are still present see more below) >> >> To reproduce the error you may checkout hector from >> http://github.com/rantav/hector, switch to the 0.7.0 branch and run: >> >> $ MAVEN_OPTS="-Xmx1G -Xms256M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC >> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 >> -XX:MaxTenuringThreshold=1" *mvn test -Dtest=CassandraClusterTest* >> >> (I'm using the default memory settings from 0.7.0) >> >> The test class source code is here >> http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java >> >> This test basically adds, removes, renames keyspaces etc. >> >> Here's the OOM log: >> >> 18:07:04,197 INFO ColumnFamilyStore:734 - Enqueuing flush of >> Memtable-Schema@72177013(3455 bytes, 3 operations) >> 18:07:04,487 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread >> Thread[FLUSH-WRITER-POOL:1,5,main] >> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java >> heap space >> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) >> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >> at >> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:637) >> Caused by: java.lang.OutOfMemoryError: Java heap space >> at >> org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151) >> at >> org.apache.cassandra.io.util.BufferedRandomAccessFile.(BufferedRandomAccessFile.java:144) >> at >> org.apache.cassandra.io.sstable.SSTableWriter.(SSTableWriter.java:61) >> at >> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151) >> at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44) >> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168) >> at >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> ... 2 more >> 18:07:04,491 INFO Memtable:150 - Writing Memtable-Schema@72177013(3455 >> bytes, 3 operations) >> 18:07:04,812 INFO CompactionManager:323 - Compacted to >> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-6-Data.db. >> 15,707 to 15,173 (~96% of original) bytes for 5 keys. Time: 860ms. >> 18:07:04,812 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread >> Thread[FLUSH-WRITER-POOL:1,5,main] >> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java >> heap space >> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) >> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >> at >> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:637) >> Caused by: java.lang.OutOfMemoryError: Java heap space >> at >> org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151) >> at >> org.apache.cassandra.io.util.BufferedRandomAccessFile.(BufferedRandomAccessFile.java:144) >> at >> org.apache.cassandra.io.sstable.SSTableWriter.(SSTableWriter.java:61) >> at >> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151) >> at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44) >> at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168) >> at >> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30) >> at >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> ... 2 more >> >> >> Here the log after I place a sleep(5000) at this line >> http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java#L88 >> The file-not-found error is even more suspicious... >> >> 18:04:28,293 ERROR AbstractCassandraDaemon:88 - Fatal exception in thread >> Thread[CompactionExecutor:1,1,main] >> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: >> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-7-Filter.db >> (No such file or directory) >> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) >> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >> at >> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87) >> at >> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:604) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) >> at java.lang.Thread.run(Thread.java:637) >> Caused by: java.io.FileNotFoundException: >> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-7-Filter.db >> (No such file or directory) >> at java.io.FileOutputStream.open(Native Method) >> at java.io.FileOutputStream.(FileOutputStream.java:179) >> at java.io.FileOutputStream.(FileOutputStream.java:70) >> at >> org.apache.cassandra.io.sstable.SSTableWriter$IndexWriter.close(SSTableWriter.java:382) >> at >> org.apache.cassandra.io.sstable.SSTableWriter.closeAndOpenReader(SSTableWriter.java:144) >> at >> org.apache.cassandra.db.CompactionManager.doCompaction(CompactionManager.java:314) >> at >> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:107) >> at >> org.apache.cassandra.db.CompactionManager$1.call(CompactionManager.java:88) >> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> at >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) >> ... 2 more >> >> >> $ java -version >> java version "1.6.0_20" >> Java(TM) SE Runtime Environment (build 1.6.0_20-b02-279-9M3165) >> Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01-279, mixed mode) >> >