incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ken Sandney <bluefl...@gmail.com>
Subject Re: 0.6.1 insert 1B rows, crashed when using py_stress
Date Tue, 20 Apr 2010 02:32:53 GMT
I am just running Cassandra on normal boxes, and grants 1GB of total 2GB to
Cassandra is reasonable I think. Can this problem be resolved by tuning the
thresholds described on this
page<http://wiki.apache.org/cassandra/MemtableThresholds> ,
or just be waiting for the 0.7 release as Brandon mentioned?

On Tue, Apr 20, 2010 at 10:15 AM, Jonathan Ellis <jbellis@gmail.com> wrote:

> Schubert, I don't know if you saw this in the other thread referencing
> your slides:
>
> It looks like the slowdown doesn't hit until after several GCs,
> although it's hard to tell since the scale is different on the GC
> graph and the insert throughput ones.
>
> Perhaps this is compaction kicking in, not GCs?  Definitely the extra
> I/O + CPU load from compaction will cause a drop in throughput.
>
> On Mon, Apr 19, 2010 at 9:06 PM, Schubert Zhang <zsongbo@gmail.com> wrote:
> > -Xmx1G is too small.
> > In my cluster, 8GB ram on each node, and I grant 6GB to cassandra.
> >
> > Please see my test @
> http://www.slideshare.net/schubertzhang/presentations
> >
> > –Memory, GC..., always to be the bottleneck and big issue of java-based
> > infrastructure software!
> >
> > References:
> > –http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
> > –https://issues.apache.org/jira/browse/CASSANDRA-896
> (LinkedBlockingQueue
> > issue, fixed in jdk-6u19)
> >
> > In fact, always when I using java-based infrastructure software, such as
> > Cassandra, Hadoop, HBase, etc, I am also pained about such memory/GC
> issue
> > finally.
> >
> > Then, we should provide higher harware with more RAM (such as 32GB~64GB),
> > more CPU cores (such as 8~16). And we still cannot control the
> > Out-Of-Memory-Error.
> >
> > I am thinking, maybe it is not right to leave the job of memory control
> to
> > JVM.
> >
> > I have a long experience in telecom and embedded software in past ten
> years,
> > where need robust programs and small RAM. I want to discuss following
> ideas
> > with the community:
> >
> > 1. Manage the memory by ourselves: allocate objects/resource (memory) at
> > initiating phase, and assign instances at runtime.
> > 2. Reject the request when be short of resource, instead of throws OOME
> and
> > exit (crash).
> >
> > 3. I know, it is not easy in java program.
> >
> > Schubert
> >
> > On Tue, Apr 20, 2010 at 9:40 AM, Ken Sandney <blueflycn@gmail.com>
> wrote:
> >>
> >> here is my JVM options, by default, I didn't modify them, from
> >> cassandra.in.sh
> >>>
> >>> # Arguments to pass to the JVM
> >>>
> >>> JVM_OPTS=" \
> >>>
> >>>         -ea \
> >>>
> >>>         -Xms128M \
> >>>
> >>>         -Xmx1G \
> >>>
> >>>         -XX:TargetSurvivorRatio=90 \
> >>>
> >>>         -XX:+AggressiveOpts \
> >>>
> >>>         -XX:+UseParNewGC \
> >>>
> >>>         -XX:+UseConcMarkSweepGC \
> >>>
> >>>         -XX:+CMSParallelRemarkEnabled \
> >>>
> >>>         -XX:+HeapDumpOnOutOfMemoryError \
> >>>
> >>>         -XX:SurvivorRatio=128 \
> >>>
> >>>         -XX:MaxTenuringThreshold=0 \
> >>>
> >>>         -Dcom.sun.management.jmxremote.port=8080 \
> >>>
> >>>         -Dcom.sun.management.jmxremote.ssl=false \
> >>>
> >>>         -Dcom.sun.management.jmxremote.authenticate=false"
> >>
> >> and my box is normal pc with 2GB ram, Intel E3200  @ 2.40GHz. By the
> way,
> >> I am using the latest Sun JDK
> >> On Tue, Apr 20, 2010 at 9:33 AM, Schubert Zhang <zsongbo@gmail.com>
> wrote:
> >>>
> >>> Seems you should configure larger jvm-heap.
> >>>
> >>> On Tue, Apr 20, 2010 at 9:32 AM, Schubert Zhang <zsongbo@gmail.com>
> >>> wrote:
> >>>>
> >>>> Please also post your jvm-heap and GC options, i.e. the seting in
> >>>> cassandra.in.sh
> >>>> And what about you node hardware?
> >>>>
> >>>> On Tue, Apr 20, 2010 at 9:22 AM, Ken Sandney <blueflycn@gmail.com>
> >>>> wrote:
> >>>>>
> >>>>> Hi
> >>>>> I am doing a insert test with 9 nodes, the command:
> >>>>>>
> >>>>>> stress.py -n 1000000000 -t 1000 -c 10 -o insert -i 5 -d
> >>>>>> 10.0.0.1,10.0.0.2.....
> >>>>>
> >>>>> and  5 of the 9 nodes were cashed, only about 6'500'000 rows were
> >>>>> inserted
> >>>>> I checked out the system.log and seems the reason are 'out of
> memory'.
> >>>>> I don't if this had something to do with my settings.
> >>>>> Any idea about this?
> >>>>> Thank you, and the following are the errors from system.log
> >>>>>
> >>>>>>
> >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 20:43:14,013
> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
> >>>>>> Thread[CACHETABLE-TIMER-1,5,main]
> >>>>>>
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76)
> >>>>>>
> >>>>>>         at java.util.TimerThread.mainLoop(Timer.java:512)
> >>>>>>
> >>>>>>         at java.util.TimerThread.run(Timer.java:462)
> >>>>>>
> >>>>>> ERROR [ROW-MUTATION-STAGE:9] 2010-04-19 20:43:27,932
> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
> >>>>>> Thread[ROW-MUTATION-STAGE:9,5,main]
> >>>>>>
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:893)
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1893)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:192)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:118)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:108)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:359)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:369)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:322)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:45)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>>>>>
> >>>>>>         at java.lang.Thread.run(Thread.java:619)
> >>>>>
> >>>>> and another
> >>>>>>
> >>>>>>  INFO [GC inspection] 2010-04-19 21:13:09,034 GCInspector.java
(line
> >>>>>> 110) GC for ConcurrentMarkSweep: 2016 ms, 1239096 reclaimed
leaving
> >>>>>> 1094238944 used; max is 1211826176
> >>>>>>
> >>>>>> ERROR [Thread-14] 2010-04-19 21:23:18,508 CassandraDaemon.java
(line
> >>>>>> 78) Fatal exception in thread Thread[Thread-14,5,main]
> >>>>>>
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>>         at
> sun.nio.ch.Util.releaseTemporaryDirectBuffer(Util.java:67)
> >>>>>>
> >>>>>>         at sun.nio.ch.IOUtil.read(IOUtil.java:212)
> >>>>>>
> >>>>>>         at
> >>>>>> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
> >>>>>>
> >>>>>>         at
> >>>>>>
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176)
> >>>>>>
> >>>>>>         at
> >>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
> >>>>>>
> >>>>>>         at java.io.InputStream.read(InputStream.java:85)
> >>>>>>
> >>>>>>         at
> >>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:64)
> >>>>>>
> >>>>>>         at java.io.DataInputStream.readInt(DataInputStream.java:370)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:70)
> >>>>>>
> >>>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:23:18,514
> >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor
> futuretask
> >>>>>>
> >>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
> >>>>>> Java heap space
> >>>>>>
> >>>>>>         at
> >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
> >>>>>>
> >>>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >>>>>>
> >>>>>>         at java.lang.Thread.run(Thread.java:619)
> >>>>>>
> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>>  INFO [FLUSH-WRITER-POOL:1] 2010-04-19 21:23:25,600 Memtable.java
> >>>>>> (line 162) Completed flushing
> >>>>>> /m/cassandra/data/Keyspace1/Standard1-623-Data.db
> >>>>>>
> >>>>>> ERROR [Thread-13] 2010-04-19 21:23:18,514 CassandraDaemon.java
(line
> >>>>>> 78) Fatal exception in thread Thread[Thread-13,5,main]
> >>>>>>
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>> ERROR [Thread-15] 2010-04-19 21:23:18,514 CassandraDaemon.java
(line
> >>>>>> 78) Fatal exception in thread Thread[Thread-15,5,main]
> >>>>>>
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:23:18,514
> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
> >>>>>> Thread[CACHETABLE-TIMER-1,5,main]
> >>>>>>
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>
> >>>>>
> >>>>> and
> >>>>>>
> >>>>>>  INFO 21:00:31,319 GC for ConcurrentMarkSweep: 1417 ms, 206216
> >>>>>> reclaimed leaving 1094527752 used; max is 1211826176
> >>>>>>
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>> Dumping heap to java_pid28670.hprof ...
> >>>>>>
> >>>>>>  INFO 21:01:23,882 GC for ConcurrentMarkSweep: 2100 ms, 734008
> >>>>>> reclaimed leaving 1093996648 used; max is 1211826176
> >>>>>>
> >>>>>> Heap dump file created [1095841554 bytes in 12.960 secs]
> >>>>>>
> >>>>>>  INFO 21:01:45,082 GC for ConcurrentMarkSweep: 2089 ms, 769968
> >>>>>> reclaimed leaving 1093960776 used; max is 1211826176
> >>>>>>
> >>>>>> ERROR 21:01:49,559 Fatal exception in thread Thread[Hint
> >>>>>> delivery,5,main]
> >>>>>>
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>>         at java.util.Arrays.copyOf(Arrays.java:2772)
> >>>>>>
> >>>>>>         at java.util.Arrays.copyOf(Arrays.java:2746)
> >>>>>>
> >>>>>>         at java.util.ArrayList.ensureCapacity(ArrayList.java:187)
> >>>>>>
> >>>>>>         at java.util.ArrayList.add(ArrayList.java:378)
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ConcurrentSkipListMap.toList(ConcurrentSkipListMap.java:2341)
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ConcurrentSkipListMap$Values.toArray(ConcurrentSkipListMap.java:2445)
> >>>>>>
> >>>>>>         at
> >>>>>> org.apache.cassandra.db.Memtable.getSliceIterator(Memtable.java:207)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.filter.SliceQueryFilter.getMemColumnIterator(SliceQueryFilter.java:58)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.filter.QueryFilter.getMemColumnIterator(QueryFilter.java:53)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:816)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:175)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >>>>>>
> >>>>>>         at java.lang.Thread.run(Thread.java:636)
> >>>>>>
> >>>>>>  INFO 21:01:56,123 GC for ConcurrentMarkSweep: 2115 ms, 893240
> >>>>>> reclaimed leaving 1093862712 used; max is 1211826176
> >>>>>
> >>>>> and
> >>>>>>
> >>>>>> ERROR [Hint delivery] 2010-04-19 21:57:07,089 CassandraDaemon.java
> >>>>>> (line 78) Fatal exception in thread Thread[Hint delivery,5,main]
> >>>>>>
> >>>>>> java.lang.RuntimeException: java.lang.RuntimeException:
> >>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
> Java
> >>>>>> heap space
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
> >>>>>>
> >>>>>>         at java.lang.Thread.run(Thread.java:636)
> >>>>>>
> >>>>>> Caused by: java.lang.RuntimeException:
> >>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
> Java
> >>>>>> heap space
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:209)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
> >>>>>>
> >>>>>>         ... 1 more
> >>>>>>
> >>>>>> Caused by: java.util.concurrent.ExecutionException:
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>>         at
> >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> >>>>>>
> >>>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:205)
> >>>>>>
> >>>>>>         ... 4 more
> >>>>>>
> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>> ERROR [MESSAGE-DESERIALIZER-POOL:1] 2010-04-19 21:57:07,089
> >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor
> futuretask
> >>>>>>
> >>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
> >>>>>> Java heap space
> >>>>>>
> >>>>>>         at
> >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> >>>>>>
> >>>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >>>>>>
> >>>>>>         at java.lang.Thread.run(Thread.java:636)
> >>>>>>
> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:57:07,089
> >>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor
> futuretask
> >>>>>>
> >>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
> >>>>>> Java heap space
> >>>>>>
> >>>>>>         at
> >>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
> >>>>>>
> >>>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:111)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
> >>>>>>
> >>>>>>         at
> >>>>>>
> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
> >>>>>>
> >>>>>>         at
> >>>>>>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> >>>>>>
> >>>>>>         at java.lang.Thread.run(Thread.java:636)
> >>>>>>
> >>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:56:29,572
> >>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
> >>>>>> Thread[CACHETABLE-TIMER-1,5,main]
> >>>>>>
> >>>>>> java.lang.OutOfMemoryError: Java heap space
> >>>>>>
> >>>>>>         at java.util.HashMap.<init>(HashMap.java:226)
       at
> >>>>>>
> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76)
> >>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)   
    at
> >>>>>> java.util.TimerThread.run(Timer.java:484)
> >>>>>
> >>>>>
> >>>
> >>
> >
> >
>

Mime
View raw message