incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ran Tavory <ran...@gmail.com>
Subject Re: OOM when running hector's unit tests at 0.7.0
Date Sat, 25 Sep 2010 11:43:06 GMT
FYI, when I increase the test memory to 1G to resemble the default setting
in cassandra the OOM doesn't happen, so that's a non-issue.

On Sat, Sep 18, 2010 at 9:02 PM, Ran Tavory <rantav@gmail.com> wrote:

> I haven't tried the python tests. What's also interesting is that if I
> run a single java test case then there's no oom. Only when I run the
> entire class then I get it. Also if I ads random sleeps then there is
> no oom but other more interesting errors surface.
>
> On Saturday, September 18, 2010, Nate McCall <nate@riptano.com> wrote:
> > Ran, have you tried to reproduce this using cassandra's python system
> > tests? Those have coverage of the system methods as well.
> >
> > On 9/18/10, Nate McCall <nate@riptano.com> wrote:
> >> An additional point - we just added the system_* methods and this test
> >> class does an add, rename then drop for both CF and keyspace.
> >>
> >> On 9/18/10, Ran Tavory <rantav@gmail.com> wrote:
> >>> I started seeing OOM when running hector's unit-tests on 0.7.0. The OOM
> >>> is
> >>> in cassandra's code so either there had been recent changes within
> >>> cassandra
> >>> or it's just that we added new coverage which causes this OOM.
> >>>
> >>> On my mac it's 100% repro but other hector devs have reported they
> don't
> >>> get
> >>> this OOM. Moreover, it I place a Thread.sleep(5000) in one of the tests
> >>> the
> >>> OOM goes away (but other errors are still present see more below)
> >>>
> >>> To reproduce the error you may checkout hector from
> >>> http://github.com/rantav/hector, switch to the 0.7.0 branch and run:
> >>>
> >>> $ MAVEN_OPTS="-Xmx1G -Xms256M -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
> >>> -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8
> >>> -XX:MaxTenuringThreshold=1" *mvn test -Dtest=CassandraClusterTest*
> >>>
> >>> (I'm using the default memory settings from 0.7.0)
> >>>
> >>> The test class source code is here
> >>>
> http://github.com/rantav/hector/blob/0.7.0/src/test/java/me/prettyprint/cassandra/service/CassandraClusterTest.java
> >>>
> >>> This test basically adds, removes, renames keyspaces etc.
> >>>
> >>> Here's the OOM log:
> >>>
> >>> 18:07:04,197  INFO ColumnFamilyStore:734 - Enqueuing flush of
> >>> Memtable-Schema@72177013(3455 bytes, 3 operations)
> >>> 18:07:04,487 ERROR AbstractCassandraDaemon:88 - Fatal exception in
> thread
> >>> Thread[FLUSH-WRITER-POOL:1,5,main]
> >>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
> Java
> >>> heap space
> >>> at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> >>>  at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >>> at
> >>>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:87)
> >>>  at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> >>> at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>>  at java.lang.Thread.run(Thread.java:637)
> >>> Caused by: java.lang.OutOfMemoryError: Java heap space
> >>> at
> >>>
> org.apache.cassandra.io.util.BufferedRandomAccessFile.init(BufferedRandomAccessFile.java:151)
> >>>  at
> >>>
> org.apache.cassandra.io.util.BufferedRandomAccessFile.<init>(BufferedRandomAccessFile.java:144)
> >>> at
> >>>
> org.apache.cassandra.io.sstable.SSTableWriter.<init>(SSTableWriter.java:61)
> >>>  at
> >>> org.apache.cassandra.db.Memtable.writeSortedContents(Memtable.java:151)
> >>> at org.apache.cassandra.db.Memtable.access$000(Memtable.java:44)
> >>>  at org.apache.cassandra.db.Memtable$1.runMayThrow(Memtable.java:168)
> >>> at
> >>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >>>  at
> >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
> >>> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> >>>  at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> >>> at
> >>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>>  ... 2 more
> >>> 18:07:04,491  INFO Memtable:150 - Writing Memtable-Schema@72177013
> (3455
> >>> bytes, 3 operations)
> >>> 18:07:04,812  INFO CompactionManager:323 - Compacted to
> >>>
> /Users/ran/dev/cassandra/hector/tmp/var/lib/cassandra/data/system/Schema-tmp-e-6-Data.db.
> >>>  15,707 to 15,173 (~96% of original) bytes for 5 keys
>
> --
> /Ran
>



-- 
/Ran

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message