incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: 0.6.1 insert 1B rows, crashed when using py_stress
Date Tue, 20 Apr 2010 02:15:47 GMT
Schubert, I don't know if you saw this in the other thread referencing
your slides:

It looks like the slowdown doesn't hit until after several GCs,
although it's hard to tell since the scale is different on the GC
graph and the insert throughput ones.

Perhaps this is compaction kicking in, not GCs?  Definitely the extra
I/O + CPU load from compaction will cause a drop in throughput.

On Mon, Apr 19, 2010 at 9:06 PM, Schubert Zhang <zsongbo@gmail.com> wrote:
> -Xmx1G is too small.
> In my cluster, 8GB ram on each node, and I grant 6GB to cassandra.
>
> Please see my test @ http://www.slideshare.net/schubertzhang/presentations
>
> –Memory, GC..., always to be the bottleneck and big issue of java-based
> infrastructure software!
>
> References:
> –http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts
> –https://issues.apache.org/jira/browse/CASSANDRA-896  (LinkedBlockingQueue
> issue, fixed in jdk-6u19)
>
> In fact, always when I using java-based infrastructure software, such as
> Cassandra, Hadoop, HBase, etc, I am also pained about such memory/GC issue
> finally.
>
> Then, we should provide higher harware with more RAM (such as 32GB~64GB),
> more CPU cores (such as 8~16). And we still cannot control the
> Out-Of-Memory-Error.
>
> I am thinking, maybe it is not right to leave the job of memory control to
> JVM.
>
> I have a long experience in telecom and embedded software in past ten years,
> where need robust programs and small RAM. I want to discuss following ideas
> with the community:
>
> 1. Manage the memory by ourselves: allocate objects/resource (memory) at
> initiating phase, and assign instances at runtime.
> 2. Reject the request when be short of resource, instead of throws OOME and
> exit (crash).
>
> 3. I know, it is not easy in java program.
>
> Schubert
>
> On Tue, Apr 20, 2010 at 9:40 AM, Ken Sandney <blueflycn@gmail.com> wrote:
>>
>> here is my JVM options, by default, I didn't modify them, from
>> cassandra.in.sh
>>>
>>> # Arguments to pass to the JVM
>>>
>>> JVM_OPTS=" \
>>>
>>>         -ea \
>>>
>>>         -Xms128M \
>>>
>>>         -Xmx1G \
>>>
>>>         -XX:TargetSurvivorRatio=90 \
>>>
>>>         -XX:+AggressiveOpts \
>>>
>>>         -XX:+UseParNewGC \
>>>
>>>         -XX:+UseConcMarkSweepGC \
>>>
>>>         -XX:+CMSParallelRemarkEnabled \
>>>
>>>         -XX:+HeapDumpOnOutOfMemoryError \
>>>
>>>         -XX:SurvivorRatio=128 \
>>>
>>>         -XX:MaxTenuringThreshold=0 \
>>>
>>>         -Dcom.sun.management.jmxremote.port=8080 \
>>>
>>>         -Dcom.sun.management.jmxremote.ssl=false \
>>>
>>>         -Dcom.sun.management.jmxremote.authenticate=false"
>>
>> and my box is normal pc with 2GB ram, Intel E3200  @ 2.40GHz. By the way,
>> I am using the latest Sun JDK
>> On Tue, Apr 20, 2010 at 9:33 AM, Schubert Zhang <zsongbo@gmail.com> wrote:
>>>
>>> Seems you should configure larger jvm-heap.
>>>
>>> On Tue, Apr 20, 2010 at 9:32 AM, Schubert Zhang <zsongbo@gmail.com>
>>> wrote:
>>>>
>>>> Please also post your jvm-heap and GC options, i.e. the seting in
>>>> cassandra.in.sh
>>>> And what about you node hardware?
>>>>
>>>> On Tue, Apr 20, 2010 at 9:22 AM, Ken Sandney <blueflycn@gmail.com>
>>>> wrote:
>>>>>
>>>>> Hi
>>>>> I am doing a insert test with 9 nodes, the command:
>>>>>>
>>>>>> stress.py -n 1000000000 -t 1000 -c 10 -o insert -i 5 -d
>>>>>> 10.0.0.1,10.0.0.2.....
>>>>>
>>>>> and  5 of the 9 nodes were cashed, only about 6'500'000 rows were
>>>>> inserted
>>>>> I checked out the system.log and seems the reason are 'out of memory'.
>>>>> I don't if this had something to do with my settings.
>>>>> Any idea about this?
>>>>> Thank you, and the following are the errors from system.log
>>>>>
>>>>>>
>>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 20:43:14,013
>>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
>>>>>> Thread[CACHETABLE-TIMER-1,5,main]
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76)
>>>>>>
>>>>>>         at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>>>
>>>>>>         at java.util.TimerThread.run(Timer.java:462)
>>>>>>
>>>>>> ERROR [ROW-MUTATION-STAGE:9] 2010-04-19 20:43:27,932
>>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
>>>>>> Thread[ROW-MUTATION-STAGE:9,5,main]
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:893)
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1893)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:192)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:118)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:108)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:359)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:369)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:322)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:45)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>
>>>>>>         at java.lang.Thread.run(Thread.java:619)
>>>>>
>>>>> and another
>>>>>>
>>>>>>  INFO [GC inspection] 2010-04-19 21:13:09,034 GCInspector.java (line
>>>>>> 110) GC for ConcurrentMarkSweep: 2016 ms, 1239096 reclaimed leaving
>>>>>> 1094238944 used; max is 1211826176
>>>>>>
>>>>>> ERROR [Thread-14] 2010-04-19 21:23:18,508 CassandraDaemon.java (line
>>>>>> 78) Fatal exception in thread Thread[Thread-14,5,main]
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>>         at sun.nio.ch.Util.releaseTemporaryDirectBuffer(Util.java:67)
>>>>>>
>>>>>>         at sun.nio.ch.IOUtil.read(IOUtil.java:212)
>>>>>>
>>>>>>         at
>>>>>> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>>>>>>
>>>>>>         at
>>>>>> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176)
>>>>>>
>>>>>>         at
>>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
>>>>>>
>>>>>>         at java.io.InputStream.read(InputStream.java:85)
>>>>>>
>>>>>>         at
>>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:64)
>>>>>>
>>>>>>         at java.io.DataInputStream.readInt(DataInputStream.java:370)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:70)
>>>>>>
>>>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:23:18,514
>>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
>>>>>>
>>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
>>>>>> Java heap space
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>>>>>>
>>>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>>>
>>>>>>         at java.lang.Thread.run(Thread.java:619)
>>>>>>
>>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>>  INFO [FLUSH-WRITER-POOL:1] 2010-04-19 21:23:25,600 Memtable.java
>>>>>> (line 162) Completed flushing
>>>>>> /m/cassandra/data/Keyspace1/Standard1-623-Data.db
>>>>>>
>>>>>> ERROR [Thread-13] 2010-04-19 21:23:18,514 CassandraDaemon.java (line
>>>>>> 78) Fatal exception in thread Thread[Thread-13,5,main]
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>> ERROR [Thread-15] 2010-04-19 21:23:18,514 CassandraDaemon.java (line
>>>>>> 78) Fatal exception in thread Thread[Thread-15,5,main]
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:23:18,514
>>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
>>>>>> Thread[CACHETABLE-TIMER-1,5,main]
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>
>>>>>
>>>>> and
>>>>>>
>>>>>>  INFO 21:00:31,319 GC for ConcurrentMarkSweep: 1417 ms, 206216
>>>>>> reclaimed leaving 1094527752 used; max is 1211826176
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>> Dumping heap to java_pid28670.hprof ...
>>>>>>
>>>>>>  INFO 21:01:23,882 GC for ConcurrentMarkSweep: 2100 ms, 734008
>>>>>> reclaimed leaving 1093996648 used; max is 1211826176
>>>>>>
>>>>>> Heap dump file created [1095841554 bytes in 12.960 secs]
>>>>>>
>>>>>>  INFO 21:01:45,082 GC for ConcurrentMarkSweep: 2089 ms, 769968
>>>>>> reclaimed leaving 1093960776 used; max is 1211826176
>>>>>>
>>>>>> ERROR 21:01:49,559 Fatal exception in thread Thread[Hint
>>>>>> delivery,5,main]
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>>         at java.util.Arrays.copyOf(Arrays.java:2772)
>>>>>>
>>>>>>         at java.util.Arrays.copyOf(Arrays.java:2746)
>>>>>>
>>>>>>         at java.util.ArrayList.ensureCapacity(ArrayList.java:187)
>>>>>>
>>>>>>         at java.util.ArrayList.add(ArrayList.java:378)
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ConcurrentSkipListMap.toList(ConcurrentSkipListMap.java:2341)
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ConcurrentSkipListMap$Values.toArray(ConcurrentSkipListMap.java:2445)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.Memtable.getSliceIterator(Memtable.java:207)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.filter.SliceQueryFilter.getMemColumnIterator(SliceQueryFilter.java:58)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.filter.QueryFilter.getMemColumnIterator(QueryFilter.java:53)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:816)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:175)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>>>>
>>>>>>         at java.lang.Thread.run(Thread.java:636)
>>>>>>
>>>>>>  INFO 21:01:56,123 GC for ConcurrentMarkSweep: 2115 ms, 893240
>>>>>> reclaimed leaving 1093862712 used; max is 1211826176
>>>>>
>>>>> and
>>>>>>
>>>>>> ERROR [Hint delivery] 2010-04-19 21:57:07,089 CassandraDaemon.java
>>>>>> (line 78) Fatal exception in thread Thread[Hint delivery,5,main]
>>>>>>
>>>>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java
>>>>>> heap space
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>>>>>
>>>>>>         at java.lang.Thread.run(Thread.java:636)
>>>>>>
>>>>>> Caused by: java.lang.RuntimeException:
>>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java
>>>>>> heap space
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:209)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>>>>
>>>>>>         ... 1 more
>>>>>>
>>>>>> Caused by: java.util.concurrent.ExecutionException:
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>>>>>>
>>>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:205)
>>>>>>
>>>>>>         ... 4 more
>>>>>>
>>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>> ERROR [MESSAGE-DESERIALIZER-POOL:1] 2010-04-19 21:57:07,089
>>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
>>>>>>
>>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
>>>>>> Java heap space
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>>>>>>
>>>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>>
>>>>>>         at java.lang.Thread.run(Thread.java:636)
>>>>>>
>>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:57:07,089
>>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
>>>>>>
>>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
>>>>>> Java heap space
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>>>>>>
>>>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>>>>>>
>>>>>>         at
>>>>>> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
>>>>>>
>>>>>>         at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>>>
>>>>>>         at java.lang.Thread.run(Thread.java:636)
>>>>>>
>>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:56:29,572
>>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
>>>>>> Thread[CACHETABLE-TIMER-1,5,main]
>>>>>>
>>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>>
>>>>>>         at java.util.HashMap.<init>(HashMap.java:226)
       at
>>>>>> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76)
>>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)    
   at
>>>>>> java.util.TimerThread.run(Timer.java:484)
>>>>>
>>>>>
>>>
>>
>
>

Mime
View raw message