incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Schubert Zhang <zson...@gmail.com>
Subject Re: 0.6.1 insert 1B rows, crashed when using py_stress
Date Tue, 20 Apr 2010 02:06:44 GMT
-Xmx1G is too small.
In my cluster, 8GB ram on each node, and I grant 6GB to cassandra.

Please see my test @ http://www.slideshare.net/schubertzhang/presentations

幻灯片 5
–Memory, GC..., always to be the bottleneck and big issue of java-based
infrastructure software!

References:
–http://wiki.apache.org/cassandra/FAQ#slows_down_after_l<http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts>
otso_inserts<http://wiki.apache.org/cassandra/FAQ#slows_down_after_lotso_inserts>
–https://issues.apache.org/jira/browse/CASSANDRA-896  (LinkedBlockingQueue
issue, fixed in jdk-6u19)


In fact, always when I using java-based infrastructure software, such as
Cassandra, Hadoop, HBase, etc, I am also pained about such memory/GC issue
finally.

Then, we should provide higher harware with more RAM (such as 32GB~64GB),
more CPU cores (such as 8~16). And we still cannot control the
Out-Of-Memory-Error.

I am thinking, maybe it is not right to leave the job of memory control to
JVM.

I have a long experience in telecom and embedded software in past ten years,
where need robust programs and small RAM. I want to discuss following ideas
with the community:

1. Manage the memory by ourselves: allocate objects/resource (memory) at
initiating phase, and assign instances at runtime.
2. Reject the request when be short of resource, instead of throws OOME and
exit (crash).

3. I know, it is not easy in java program.

Schubert

On Tue, Apr 20, 2010 at 9:40 AM, Ken Sandney <blueflycn@gmail.com> wrote:

> here is my JVM options, by default, I didn't modify them, from
> cassandra.in.sh
>
> # Arguments to pass to the JVM
>
> JVM_OPTS=" \
>
>         -ea \
>
>         -Xms128M \
>
>         -Xmx1G \
>
>         -XX:TargetSurvivorRatio=90 \
>
>         -XX:+AggressiveOpts \
>
>         -XX:+UseParNewGC \
>
>         -XX:+UseConcMarkSweepGC \
>
>         -XX:+CMSParallelRemarkEnabled \
>
>         -XX:+HeapDumpOnOutOfMemoryError \
>
>         -XX:SurvivorRatio=128 \
>
>         -XX:MaxTenuringThreshold=0 \
>
>         -Dcom.sun.management.jmxremote.port=8080 \
>
>         -Dcom.sun.management.jmxremote.ssl=false \
>
>         -Dcom.sun.management.jmxremote.authenticate=false"
>
>
> and my box is normal pc with 2GB ram, Intel E3200  @ 2.40GHz. By the way, I
> am using the latest Sun JDK
>
> On Tue, Apr 20, 2010 at 9:33 AM, Schubert Zhang <zsongbo@gmail.com> wrote:
>
>> Seems you should configure larger jvm-heap.
>>
>>
>> On Tue, Apr 20, 2010 at 9:32 AM, Schubert Zhang <zsongbo@gmail.com>wrote:
>>
>>> Please also post your jvm-heap and GC options, i.e. the seting in
>>> cassandra.in.sh
>>> And what about you node hardware?
>>>
>>> On Tue, Apr 20, 2010 at 9:22 AM, Ken Sandney <blueflycn@gmail.com>wrote:
>>>
>>>> Hi
>>>> I am doing a insert test with 9 nodes, the command:
>>>>
>>>>> stress.py -n 1000000000 -t 1000 -c 10 -o insert -i 5 -d
>>>>> 10.0.0.1,10.0.0.2.....
>>>>
>>>> and  5 of the 9 nodes were cashed, only about 6'500'000 rows were
>>>> inserted
>>>> I checked out the system.log and seems the reason are 'out of memory'. I
>>>> don't if this had something to do with my settings.
>>>> Any idea about this?
>>>> Thank you, and the following are the errors from system.log
>>>>
>>>>
>>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 20:43:14,013 CassandraDaemon.java
>>>>> (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main]
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>>>>         at
>>>>> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76)
>>>>
>>>>         at java.util.TimerThread.mainLoop(Timer.java:512)
>>>>
>>>>         at java.util.TimerThread.run(Timer.java:462)
>>>>
>>>> ERROR [ROW-MUTATION-STAGE:9] 2010-04-19 20:43:27,932
>>>>> CassandraDaemon.java (line 78) Fatal exception in thread
>>>>> Thread[ROW-MUTATION-STAGE:9,5,main]
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>>>>         at
>>>>> java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:893)
>>>>
>>>>         at
>>>>> java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1893)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:192)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(ColumnFamilySerializer.java:118)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFamilySerializer.java:108)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMutation.java:359)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:369)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation.java:322)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerbHandler.java:45)
>>>>
>>>>         at
>>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
>>>>
>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>>>
>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>
>>>>         at java.lang.Thread.run(Thread.java:619)
>>>>
>>>>
>>>> and another
>>>>
>>>>  INFO [GC inspection] 2010-04-19 21:13:09,034 GCInspector.java (line
>>>>> 110) GC for ConcurrentMarkSweep: 2016 ms, 1239096 reclaimed leaving
>>>>> 1094238944 used; max is 1211826176
>>>>
>>>> ERROR [Thread-14] 2010-04-19 21:23:18,508 CassandraDaemon.java (line 78)
>>>>> Fatal exception in thread Thread[Thread-14,5,main]
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>>>>         at sun.nio.ch.Util.releaseTemporaryDirectBuffer(Util.java:67)
>>>>
>>>>         at sun.nio.ch.IOUtil.read(IOUtil.java:212)
>>>>
>>>>         at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
>>>>
>>>>         at
>>>>> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:176)
>>>>
>>>>         at
>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86)
>>>>
>>>>         at java.io.InputStream.read(InputStream.java:85)
>>>>
>>>>         at
>>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:64)
>>>>
>>>>         at java.io.DataInputStream.readInt(DataInputStream.java:370)
>>>>
>>>>         at
>>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:70)
>>>>
>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:23:18,514
>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
>>>>
>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
>>>>> Java heap space
>>>>
>>>>         at
>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
>>>>
>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>>>>
>>>>         at
>>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
>>>>
>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:888)
>>>>
>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>>>
>>>>         at java.lang.Thread.run(Thread.java:619)
>>>>
>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>
>>>>  INFO [FLUSH-WRITER-POOL:1] 2010-04-19 21:23:25,600 Memtable.java (line
>>>>> 162) Completed flushing /m/cassandra/data/Keyspace1/Standard1-623-Data.db
>>>>
>>>> ERROR [Thread-13] 2010-04-19 21:23:18,514 CassandraDaemon.java (line 78)
>>>>> Fatal exception in thread Thread[Thread-13,5,main]
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>>>> ERROR [Thread-15] 2010-04-19 21:23:18,514 CassandraDaemon.java (line 78)
>>>>> Fatal exception in thread Thread[Thread-15,5,main]
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:23:18,514 CassandraDaemon.java
>>>>> (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main]
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>>>>
>>>> and
>>>>
>>>>  INFO 21:00:31,319 GC for ConcurrentMarkSweep: 1417 ms, 206216 reclaimed
>>>>> leaving 1094527752 used; max is 1211826176
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>>>> Dumping heap to java_pid28670.hprof ...
>>>>
>>>>  INFO 21:01:23,882 GC for ConcurrentMarkSweep: 2100 ms, 734008 reclaimed
>>>>> leaving 1093996648 used; max is 1211826176
>>>>
>>>> Heap dump file created [1095841554 bytes in 12.960 secs]
>>>>
>>>>  INFO 21:01:45,082 GC for ConcurrentMarkSweep: 2089 ms, 769968 reclaimed
>>>>> leaving 1093960776 used; max is 1211826176
>>>>
>>>> ERROR 21:01:49,559 Fatal exception in thread Thread[Hint
>>>>> delivery,5,main]
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>>>>         at java.util.Arrays.copyOf(Arrays.java:2772)
>>>>
>>>>         at java.util.Arrays.copyOf(Arrays.java:2746)
>>>>
>>>>         at java.util.ArrayList.ensureCapacity(ArrayList.java:187)
>>>>
>>>>         at java.util.ArrayList.add(ArrayList.java:378)
>>>>
>>>>         at
>>>>> java.util.concurrent.ConcurrentSkipListMap.toList(ConcurrentSkipListMap.java:2341)
>>>>
>>>>         at
>>>>> java.util.concurrent.ConcurrentSkipListMap$Values.toArray(ConcurrentSkipListMap.java:2445)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.Memtable.getSliceIterator(Memtable.java:207)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.filter.SliceQueryFilter.getMemColumnIterator(SliceQueryFilter.java:58)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.filter.QueryFilter.getMemColumnIterator(QueryFilter.java:53)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:816)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:750)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:719)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:175)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100)
>>>>
>>>>         at
>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>>
>>>>         at java.lang.Thread.run(Thread.java:636)
>>>>
>>>>  INFO 21:01:56,123 GC for ConcurrentMarkSweep: 2115 ms, 893240 reclaimed
>>>>> leaving 1093862712 used; max is 1211826176
>>>>
>>>>
>>>> and
>>>>
>>>> ERROR [Hint delivery] 2010-04-19 21:57:07,089 CassandraDaemon.java (line
>>>>> 78) Fatal exception in thread Thread[Hint delivery,5,main]
>>>>
>>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java
>>>>> heap space
>>>>
>>>>         at
>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:34)
>>>>
>>>>         at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Caused by: java.lang.RuntimeException:
>>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java
>>>>> heap space
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:209)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOffManager.java:80)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHandOffManager.java:100)
>>>>
>>>>         at
>>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>>
>>>>         ... 1 more
>>>>
>>>> Caused by: java.util.concurrent.ExecutionException:
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>>>>         at
>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>>>>
>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHandOffManager.java:205)
>>>>
>>>>         ... 4 more
>>>>
>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>
>>>> ERROR [MESSAGE-DESERIALIZER-POOL:1] 2010-04-19 21:57:07,089
>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
>>>>
>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
>>>>> Java heap space
>>>>
>>>>         at
>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>>>>
>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>>>>
>>>>         at
>>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>>>>
>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
>>>>
>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>
>>>>         at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>
>>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:57:07,089
>>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuretask
>>>>
>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
>>>>> Java heap space
>>>>
>>>>         at
>>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252)
>>>>
>>>>         at java.util.concurrent.FutureTask.get(FutureTask.java:111)
>>>>
>>>>         at
>>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
>>>>
>>>>         at
>>>>> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExecute(CompactionManager.java:582)
>>>>
>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1118)
>>>>
>>>>         at
>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
>>>>
>>>>         at java.lang.Thread.run(Thread.java:636)
>>>>
>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>
>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:56:29,572 CassandraDaemon.java
>>>>> (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main]
>>>>
>>>> java.lang.OutOfMemoryError: Java heap space
>>>>
>>>>         at java.util.HashMap.<init>(HashMap.java:226)        at
>>>>> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.java:76)
>>>>>        at java.util.TimerThread.mainLoop(Timer.java:534)        at
>>>>> java.util.TimerThread.run(Timer.java:484)
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>

Mime
View raw message