incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Sandney <bluefl...@gmail.com>
Subject Re: 0.6.1 insert 1B rows, crashed when using py_stress
Date Tue, 20 Apr 2010 02:39:52 GMT
Sorry I just don't know how to resolve this :)

On Tue, Apr 20, 2010 at 10:37 AM, Jonathan Ellis <jbellis@gmail.com> wrote:

> Ken, I linked you to the FAQ answering your problem in the first reply
> you got.  Please don't hijack my replies to other people; that's rude.
>
> On Mon, Apr 19, 2010 at 9:32 PM, Ken Sandney <blueflycn@gmail.com> wrote:
> > I am just running Cassandra on normal boxes, and grants 1GB of total 2GB
> to
> > Cassandra is reasonable I think. Can this problem be resolved by tuning
> the
> > thresholds described on this page , or just be waiting for the 0.7
> release
> > as Brandon mentioned?
> >
> > On Tue, Apr 20, 2010 at 10:15 AM, Jonathan Ellis <jbellis@gmail.com>
> wrote:
> >>
> >> Schubert, I don't know if you saw this in the other thread referencing
> >> your slides:
> >>
> >> It looks like the slowdown doesn't hit until after several GCs,
> >> although it's hard to tell since the scale is different on the GC
> >> graph and the insert throughput ones.
> >>
> >> Perhaps this is compaction kicking in, not GCs?  Definitely the extra
> >> I/O + CPU load from compaction will cause a drop in throughput.
> >>
> >> On Mon, Apr 19, 2010 at 9:06 PM, Schubert Zhang <zsongbo@gmail.com>
> wrote:
> >> > -Xmx1G is too small.
> >> > In my cluster, 8GB ram on each node, and I grant 6GB to cassandra.
> >> >
> >> > Please see my test @
> >> > http://www.slideshare.net/schubertzhang/presentations
> >> >
> >> > –Memory, GC..., always to be the bottleneck and big issue of
> java-based
> >> > infrastructure software!
> >> >
> >> > References:
> >> > –http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
> >> > –https://issues.apache.org/jira/browse/CASSANDRA-896
> >> > (LinkedBlockingQueue
> >> > issue, fixed in jdk-6u19)
> >> >
> >> > In fact, always when I using java-based infrastructure software, such
> as
> >> > Cassandra, Hadoop, HBase, etc, I am also pained about such memory/GC
> >> > issue
> >> > finally.
> >> >
> >> > Then, we should provide higher harware with more RAM (such as
> >> > 32GB~64GB),
> >> > more CPU cores (such as 8~16). And we still cannot control the
> >> > Out-Of-Memory-Error.
> >> >
> >> > I am thinking, maybe it is not right to leave the job of memory
> control
> >> > to
> >> > JVM.
> >> >
> >> > I have a long experience in telecom and embedded software in past ten
> >> > years,
> >> > where need robust programs and small RAM. I want to discuss following
> >> > ideas
> >> > with the community:
> >> >
> >> > 1. Manage the memory by ourselves: allocate objects/resource (memory)
> at
> >> > initiating phase, and assign instances at runtime.
> >> > 2. Reject the request when be short of resource, instead of throws
> OOME
> >> > and
> >> > exit (crash).
> >> >
> >> > 3. I know, it is not easy in java program.
> >> >
> >> > Schubert
> >> >
> >> > On Tue, Apr 20, 2010 at 9:40 AM, Ken Sandney <blueflycn@gmail.com>
> >> > wrote:
> >> >>
> >> >> here is my JVM options, by default, I didn't modify them, from
> >> >> cassandra.in.sh
> >> >>>
> >> >>> # Arguments to pass to the JVM
> >> >>>
> >> >>> JVM_OPTS=" \
> >> >>>
> >> >>>         -ea \
> >> >>>
> >> >>>         -Xms128M \
> >> >>>
> >> >>>         -Xmx1G \
> >> >>>
> >> >>>         -XX:TargetSurvivorRatio=90 \
> >> >>>
> >> >>>         -XX:+AggressiveOpts \
> >> >>>
> >> >>>         -XX:+UseParNewGC \
> >> >>>
> >> >>>         -XX:+UseConcMarkSweepGC \
> >> >>>
> >> >>>         -XX:+CMSParallelRemarkEnabled \
> >> >>>
> >> >>>         -XX:+HeapDumpOnOutOfMemoryError \
> >> >>>
> >> >>>         -XX:SurvivorRatio=128 \
> >> >>>
> >> >>>         -XX:MaxTenuringThreshold=0 \
> >> >>>
> >> >>>         -Dcom.sun.management.jmxremote.port=8080 \
> >> >>>
> >> >>>         -Dcom.sun.management.jmxremote.ssl=false \
> >> >>>
> >> >>>         -Dcom.sun.management.jmxremote.authenticate=false"
> >> >>
> >> >> and my box is normal pc with 2GB ram, Intel E3200  @ 2.40GHz. By the
> >> >> way,
> >> >> I am using the latest Sun JDK
> >> >> On Tue, Apr 20, 2010 at 9:33 AM, Schubert Zhang <zsongbo@gmail.com>
> >> >> wrote:
> >> >>>
> >> >>> Seems you should configure larger jvm-heap.
> >> >>>
> >> >>> On Tue, Apr 20, 2010 at 9:32 AM, Schubert Zhang <zsongbo@gmail.com>
> >> >>> wrote:
> >> >>>>
> >> >>>> Please also post your jvm-heap and GC options, i.e. the seting
in
> >> >>>> cassandra.in.sh
> >> >>>> And what about you node hardware?
> >> >>>>
> >> >>>> On Tue, Apr 20, 2010 at 9:22 AM, Ken Sandney <blueflycn@gmail.com>
> >> >>>> wrote:
> >> >>>>>
> >> >>>>> Hi
> >> >>>>> I am doing a insert test with 9 nodes, the command:
> >> >>>>>>
> >> >>>>>> stress.py -n 1000000000 -t 1000 -c 10 -o insert -i
5 -d
> >> >>>>>> 10.0.0.1,10.0.0.2.....
> >> >>>>>
> >> >>>>> and  5 of the 9 nodes were cashed, only about 6'500'000
rows were
> >> >>>>> inserted
> >> >>>>> I checked out the system.log and seems the reason are 'out
of
> >> >>>>> memory'.
> >> >>>>> I don't if this had something to do with my settings.
> >> >>>>> Any idea about this?
> >> >>>>> Thank you, and the following are the errors from system.log
> >> >>>>>
> >> >>>>>>
> >> >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 20:43:14,013
> >> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
> >> >>>>>> Thread[CACHETABLE-TIMER-1,5,main]
> >> >>>>>>
> >> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76)
> >> >>>>>>
> >> >>>>>>         at java.util.TimerThread.mainLoop(Timer.java:512)
> >> >>>>>>
> >> >>>>>>         at java.util.TimerThread.run(Timer.java:462)
> >> >>>>>>
> >> >>>>>> ERROR [ROW-MUTATION-STAGE:9] 2010-04-19 20:43:27,932
> >> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
> >> >>>>>> Thread[ROW-MUTATION-STAGE:9,5,main]
> >> >>>>>>
> >> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:893)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1893)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:192)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:118)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:108)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:359)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:369)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:322)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:45)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >>>>>>
> >> >>>>>>         at java.lang.Thread.run(Thread.java:619)
> >> >>>>>
> >> >>>>> and another
> >> >>>>>>
> >> >>>>>>  INFO [GC inspection] 2010-04-19 21:13:09,034 GCInspector.java
> >> >>>>>> (line
> >> >>>>>> 110) GC for ConcurrentMarkSweep: 2016 ms, 1239096 reclaimed
> leaving
> >> >>>>>> 1094238944 used; max is 1211826176
> >> >>>>>>
> >> >>>>>> ERROR [Thread-14] 2010-04-19 21:23:18,508 CassandraDaemon.java
> >> >>>>>> (line
> >> >>>>>> 78) Fatal exception in thread Thread[Thread-14,5,main]
> >> >>>>>>
> >> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>> sun.nio.ch.Util.releaseTemporaryDirectBuffer(Util.java:67)
> >> >>>>>>
> >> >>>>>>         at sun.nio.ch.IOUtil.read(IOUtil.java:212)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
> >> >>>>>>
> >> >>>>>>         at java.io.InputStream.read(InputStream.java:85)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:64)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>> java.io.DataInputStream.readInt(DataInputStream.java:370)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:70)
> >> >>>>>>
> >> >>>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:23:18,514
> >> >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in
executor
> >> >>>>>> futuretask
> >> >>>>>>
> >> >>>>>> java.util.concurrent.ExecutionException:
> >> >>>>>> java.lang.OutOfMemoryError:
> >> >>>>>> Java heap space
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> >> >>>>>>
> >> >>>>>>         at
> java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> >>>>>>
> >> >>>>>>         at java.lang.Thread.run(Thread.java:619)
> >> >>>>>>
> >> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>>  INFO [FLUSH-WRITER-POOL:1] 2010-04-19 21:23:25,600
Memtable.java
> >> >>>>>> (line 162) Completed flushing
> >> >>>>>> /m/cassandra/data/Keyspace1/Standard1-623-Data.db
> >> >>>>>>
> >> >>>>>> ERROR [Thread-13] 2010-04-19 21:23:18,514 CassandraDaemon.java
> >> >>>>>> (line
> >> >>>>>> 78) Fatal exception in thread Thread[Thread-13,5,main]
> >> >>>>>>
> >> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>> ERROR [Thread-15] 2010-04-19 21:23:18,514 CassandraDaemon.java
> >> >>>>>> (line
> >> >>>>>> 78) Fatal exception in thread Thread[Thread-15,5,main]
> >> >>>>>>
> >> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:23:18,514
> >> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
> >> >>>>>> Thread[CACHETABLE-TIMER-1,5,main]
> >> >>>>>>
> >> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >> >>>>>
> >> >>>>>
> >> >>>>> and
> >> >>>>>>
> >> >>>>>>  INFO 21:00:31,319 GC for ConcurrentMarkSweep: 1417
ms, 206216
> >> >>>>>> reclaimed leaving 1094527752 used; max is 1211826176
> >> >>>>>>
> >> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>> Dumping heap to java_pid28670.hprof ...
> >> >>>>>>
> >> >>>>>>  INFO 21:01:23,882 GC for ConcurrentMarkSweep: 2100
ms, 734008
> >> >>>>>> reclaimed leaving 1093996648 used; max is 1211826176
> >> >>>>>>
> >> >>>>>> Heap dump file created [1095841554 bytes in 12.960
secs]
> >> >>>>>>
> >> >>>>>>  INFO 21:01:45,082 GC for ConcurrentMarkSweep: 2089
ms, 769968
> >> >>>>>> reclaimed leaving 1093960776 used; max is 1211826176
> >> >>>>>>
> >> >>>>>> ERROR 21:01:49,559 Fatal exception in thread Thread[Hint
> >> >>>>>> delivery,5,main]
> >> >>>>>>
> >> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>>         at java.util.Arrays.copyOf(Arrays.java:2772)
> >> >>>>>>
> >> >>>>>>         at java.util.Arrays.copyOf(Arrays.java:2746)
> >> >>>>>>
> >> >>>>>>         at java.util.ArrayList.ensureCapacity(ArrayList.java:187)
> >> >>>>>>
> >> >>>>>>         at java.util.ArrayList.add(ArrayList.java:378)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ConcurrentSkipListMap.toList(ConcurrentSkipListMap.java:2341)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ConcurrentSkipListMap$Values.toArray(ConcurrentSkipListMap.java:2445)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.Memtable.getSliceIterator(Memtable.java:207)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.filter.SliceQueryFilter.getMemColumnIterator(SliceQueryFilter.java:58)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.filter.QueryFilter.getMemColumnIterator(QueryFilter.java:53)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:816)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:175)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >> >>>>>>
> >> >>>>>>         at java.lang.Thread.run(Thread.java:636)
> >> >>>>>>
> >> >>>>>>  INFO 21:01:56,123 GC for ConcurrentMarkSweep: 2115
ms, 893240
> >> >>>>>> reclaimed leaving 1093862712 used; max is 1211826176
> >> >>>>>
> >> >>>>> and
> >> >>>>>>
> >> >>>>>> ERROR [Hint delivery] 2010-04-19 21:57:07,089
> CassandraDaemon.java
> >> >>>>>> (line 78) Fatal exception in thread Thread[Hint delivery,5,main]
> >> >>>>>>
> >> >>>>>> java.lang.RuntimeException: java.lang.RuntimeException:
> >> >>>>>> java.util.concurrent.ExecutionException:
> >> >>>>>> java.lang.OutOfMemoryError: Java
> >> >>>>>> heap space
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> >> >>>>>>
> >> >>>>>>         at java.lang.Thread.run(Thread.java:636)
> >> >>>>>>
> >> >>>>>> Caused by: java.lang.RuntimeException:
> >> >>>>>> java.util.concurrent.ExecutionException:
> >> >>>>>> java.lang.OutOfMemoryError: Java
> >> >>>>>> heap space
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:209)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >> >>>>>>
> >> >>>>>>         ... 1 more
> >> >>>>>>
> >> >>>>>> Caused by: java.util.concurrent.ExecutionException:
> >> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> >> >>>>>>
> >> >>>>>>         at
> java.util.concurrent.FutureTask.get(FutureTask.java:111)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:205)
> >> >>>>>>
> >> >>>>>>         ... 4 more
> >> >>>>>>
> >> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>> ERROR [MESSAGE-DESERIALIZER-POOL:1] 2010-04-19 21:57:07,089
> >> >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in
executor
> >> >>>>>> futuretask
> >> >>>>>>
> >> >>>>>> java.util.concurrent.ExecutionException:
> >> >>>>>> java.lang.OutOfMemoryError:
> >> >>>>>> Java heap space
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> >> >>>>>>
> >> >>>>>>         at
> java.util.concurrent.FutureTask.get(FutureTask.java:111)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >> >>>>>>
> >> >>>>>>         at java.lang.Thread.run(Thread.java:636)
> >> >>>>>>
> >> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:57:07,089
> >> >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in
executor
> >> >>>>>> futuretask
> >> >>>>>>
> >> >>>>>> java.util.concurrent.ExecutionException:
> >> >>>>>> java.lang.OutOfMemoryError:
> >> >>>>>> Java heap space
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> >> >>>>>>
> >> >>>>>>         at
> java.util.concurrent.FutureTask.get(FutureTask.java:111)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
> >> >>>>>>
> >> >>>>>>         at
> >> >>>>>>
> >> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >> >>>>>>
> >> >>>>>>         at java.lang.Thread.run(Thread.java:636)
> >> >>>>>>
> >> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:56:29,572
> >> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
> >> >>>>>> Thread[CACHETABLE-TIMER-1,5,main]
> >> >>>>>>
> >> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >> >>>>>>
> >> >>>>>>         at java.util.HashMap.<init>(HashMap.java:226)
       at
> >> >>>>>>
> >> >>>>>>
> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76)
> >> >>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)
>  at
> >> >>>>>> java.util.TimerThread.run(Timer.java:484)
> >> >>>>>
> >> >>>>>
> >> >>>
> >> >>
> >> >
> >> >
> >
> >
>

Mime
View raw message