Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 8224 invoked from network); 20 Apr 2010 02:07:12 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 20 Apr 2010 02:07:12 -0000 Received: (qmail 55996 invoked by uid 500); 20 Apr 2010 02:07:11 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 55953 invoked by uid 500); 20 Apr 2010 02:07:10 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 55945 invoked by uid 99); 20 Apr 2010 02:07:10 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 02:07:10 +0000 X-ASF-Spam-Status: No, hits=0.6 required=10.0 tests=AWL,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of zsongbo@gmail.com designates 209.85.221.192 as permitted sender) Received: from [209.85.221.192] (HELO mail-qy0-f192.google.com) (209.85.221.192) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 20 Apr 2010 02:07:06 +0000 Received: by qyk30 with SMTP id 30so4387327qyk.16 for ; Mon, 19 Apr 2010 19:06:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:content-type; bh=t/GBuhG+DYJnECy3MbbXKY1NueeZrntH+7qI+Jwetjk=; b=M9pVe5reVpAbd0mg8x6bB9Zp79zOMTUH02WQ8sD5n7lSdCjLlzqAYoNDrjW4PBQKy+ ro2x6QQp1+W9aYaW6HxoJ4u1l78HFISnld4+XpmYY6e2sy9arolUyPdfXalrCqOvhneF xLGQwxWmhzPme6vO+wzhuGK4+sOCOtqlg1FUU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=NYpbg2IfYcqQULuAC8ds8syvxO6ZYjoj7xwayinOac6zmtpJG4CWIF/NZf/1u/mR94 +lT69itWl/mfgczRBTYxfKrPZrqp0R5Y8fMpBRmfkNF0mUeXrKG/UwzYnNoElcRbo83g mdqg4WSShtI6MoFH9PCWcap2fZ8BW+sFi2j+s= MIME-Version: 1.0 Received: by 10.229.91.137 with HTTP; Mon, 19 Apr 2010 19:06:44 -0700 (PDT) In-Reply-To: References: Date: Tue, 20 Apr 2010 10:06:44 +0800 Received: by 10.229.230.65 with SMTP id jl1mr4678105qcb.7.1271729204204; Mon, 19 Apr 2010 19:06:44 -0700 (PDT) Message-ID: Subject: Re: 0.6.1 insert 1B rows, crashed when using py_stress From: Schubert Zhang To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=00163630e9a57794d00484a18704 --00163630e9a57794d00484a18704 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable -Xmx1G is too small. In my cluster, 8GB ram on each node, and I grant 6GB to cassandra. Please see my test @ http://www.slideshare.net/schubertzhang/presentations =E5=B9=BB=E7=81=AF=E7=89=87 5 =E2=80=93Memory, GC..., always to be the bottleneck and big issue of java-b= ased infrastructure software! References: =E2=80=93http://wiki.apache.org/cassandra/FAQ#slows_down_after_l otso_inserts =E2=80=93https://issues.apache.org/jira/browse/CASSANDRA-896 (LinkedBlocki= ngQueue issue, fixed in jdk-6u19) In fact, always when I using java-based infrastructure software, such as Cassandra, Hadoop, HBase, etc, I am also pained about such memory/GC issue finally. Then, we should provide higher harware with more RAM (such as 32GB~64GB), more CPU cores (such as 8~16). And we still cannot control the Out-Of-Memory-Error. I am thinking, maybe it is not right to leave the job of memory control to JVM. I have a long experience in telecom and embedded software in past ten years= , where need robust programs and small RAM. I want to discuss following ideas with the community: 1. Manage the memory by ourselves: allocate objects/resource (memory) at initiating phase, and assign instances at runtime. 2. Reject the request when be short of resource, instead of throws OOME and exit (crash). 3. I know, it is not easy in java program. Schubert On Tue, Apr 20, 2010 at 9:40 AM, Ken Sandney wrote: > here is my JVM options, by default, I didn't modify them, from > cassandra.in.sh > > # Arguments to pass to the JVM > > JVM_OPTS=3D" \ > > -ea \ > > -Xms128M \ > > -Xmx1G \ > > -XX:TargetSurvivorRatio=3D90 \ > > -XX:+AggressiveOpts \ > > -XX:+UseParNewGC \ > > -XX:+UseConcMarkSweepGC \ > > -XX:+CMSParallelRemarkEnabled \ > > -XX:+HeapDumpOnOutOfMemoryError \ > > -XX:SurvivorRatio=3D128 \ > > -XX:MaxTenuringThreshold=3D0 \ > > -Dcom.sun.management.jmxremote.port=3D8080 \ > > -Dcom.sun.management.jmxremote.ssl=3Dfalse \ > > -Dcom.sun.management.jmxremote.authenticate=3Dfalse" > > > and my box is normal pc with 2GB ram, Intel E3200 @ 2.40GHz. By the way,= I > am using the latest Sun JDK > > On Tue, Apr 20, 2010 at 9:33 AM, Schubert Zhang wrote= : > >> Seems you should configure larger jvm-heap. >> >> >> On Tue, Apr 20, 2010 at 9:32 AM, Schubert Zhang wrote= : >> >>> Please also post your jvm-heap and GC options, i.e. the seting in >>> cassandra.in.sh >>> And what about you node hardware? >>> >>> On Tue, Apr 20, 2010 at 9:22 AM, Ken Sandney wrote= : >>> >>>> Hi >>>> I am doing a insert test with 9 nodes, the command: >>>> >>>>> stress.py -n 1000000000 -t 1000 -c 10 -o insert -i 5 -d >>>>> 10.0.0.1,10.0.0.2..... >>>> >>>> and 5 of the 9 nodes were cashed, only about 6'500'000 rows were >>>> inserted >>>> I checked out the system.log and seems the reason are 'out of memory'.= I >>>> don't if this had something to do with my settings. >>>> Any idea about this? >>>> Thank you, and the following are the errors from system.log >>>> >>>> >>>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 20:43:14,013 CassandraDaemon.ja= va >>>>> (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main] >>>> >>>> java.lang.OutOfMemoryError: Java heap space >>>> >>>> at >>>>> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.j= ava:76) >>>> >>>> at java.util.TimerThread.mainLoop(Timer.java:512) >>>> >>>> at java.util.TimerThread.run(Timer.java:462) >>>> >>>> ERROR [ROW-MUTATION-STAGE:9] 2010-04-19 20:43:27,932 >>>>> CassandraDaemon.java (line 78) Fatal exception in thread >>>>> Thread[ROW-MUTATION-STAGE:9,5,main] >>>> >>>> java.lang.OutOfMemoryError: Java heap space >>>> >>>> at >>>>> java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMa= p.java:893) >>>> >>>> at >>>>> java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkip= ListMap.java:1893) >>>> >>>> at >>>>> org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:192) >>>> >>>> at >>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumns(Col= umnFamilySerializer.java:118) >>>> >>>> at >>>>> org.apache.cassandra.db.ColumnFamilySerializer.deserialize(ColumnFami= lySerializer.java:108) >>>> >>>> at >>>>> org.apache.cassandra.db.RowMutationSerializer.defreezeTheMaps(RowMuta= tion.java:359) >>>> >>>> at >>>>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation= .java:369) >>>> >>>> at >>>>> org.apache.cassandra.db.RowMutationSerializer.deserialize(RowMutation= .java:322) >>>> >>>> at >>>>> org.apache.cassandra.db.RowMutationVerbHandler.doVerb(RowMutationVerb= Handler.java:45) >>>> >>>> at >>>>> org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.= java:40) >>>> >>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec= utor.java:886) >>>> >>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:908) >>>> >>>> at java.lang.Thread.run(Thread.java:619) >>>> >>>> >>>> and another >>>> >>>> INFO [GC inspection] 2010-04-19 21:13:09,034 GCInspector.java (line >>>>> 110) GC for ConcurrentMarkSweep: 2016 ms, 1239096 reclaimed leaving >>>>> 1094238944 used; max is 1211826176 >>>> >>>> ERROR [Thread-14] 2010-04-19 21:23:18,508 CassandraDaemon.java (line 7= 8) >>>>> Fatal exception in thread Thread[Thread-14,5,main] >>>> >>>> java.lang.OutOfMemoryError: Java heap space >>>> >>>> at sun.nio.ch.Util.releaseTemporaryDirectBuffer(Util.java:67) >>>> >>>> at sun.nio.ch.IOUtil.read(IOUtil.java:212) >>>> >>>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:23= 6) >>>> >>>> at >>>>> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:17= 6) >>>> >>>> at >>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:86) >>>> >>>> at java.io.InputStream.read(InputStream.java:85) >>>> >>>> at >>>>> sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:64) >>>> >>>> at java.io.DataInputStream.readInt(DataInputStream.java:370) >>>> >>>> at >>>>> org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnect= ion.java:70) >>>> >>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:23:18,514 >>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuret= ask >>>> >>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: >>>>> Java heap space >>>> >>>> at >>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222) >>>> >>>> at java.util.concurrent.FutureTask.get(FutureTask.java:83) >>>> >>>> at >>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExe= cute(DebuggableThreadPoolExecutor.java:86) >>>> >>>> at >>>>> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExe= cute(CompactionManager.java:582) >>>> >>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec= utor.java:888) >>>> >>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:908) >>>> >>>> at java.lang.Thread.run(Thread.java:619) >>>> >>>> Caused by: java.lang.OutOfMemoryError: Java heap space >>>> >>>> INFO [FLUSH-WRITER-POOL:1] 2010-04-19 21:23:25,600 Memtable.java (lin= e >>>>> 162) Completed flushing /m/cassandra/data/Keyspace1/Standard1-623-Dat= a.db >>>> >>>> ERROR [Thread-13] 2010-04-19 21:23:18,514 CassandraDaemon.java (line 7= 8) >>>>> Fatal exception in thread Thread[Thread-13,5,main] >>>> >>>> java.lang.OutOfMemoryError: Java heap space >>>> >>>> ERROR [Thread-15] 2010-04-19 21:23:18,514 CassandraDaemon.java (line 7= 8) >>>>> Fatal exception in thread Thread[Thread-15,5,main] >>>> >>>> java.lang.OutOfMemoryError: Java heap space >>>> >>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:23:18,514 CassandraDaemon.jav= a >>>>> (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main] >>>> >>>> java.lang.OutOfMemoryError: Java heap space >>>> >>>> >>>> and >>>> >>>> INFO 21:00:31,319 GC for ConcurrentMarkSweep: 1417 ms, 206216 reclaim= ed >>>>> leaving 1094527752 used; max is 1211826176 >>>> >>>> java.lang.OutOfMemoryError: Java heap space >>>> >>>> Dumping heap to java_pid28670.hprof ... >>>> >>>> INFO 21:01:23,882 GC for ConcurrentMarkSweep: 2100 ms, 734008 reclaim= ed >>>>> leaving 1093996648 used; max is 1211826176 >>>> >>>> Heap dump file created [1095841554 bytes in 12.960 secs] >>>> >>>> INFO 21:01:45,082 GC for ConcurrentMarkSweep: 2089 ms, 769968 reclaim= ed >>>>> leaving 1093960776 used; max is 1211826176 >>>> >>>> ERROR 21:01:49,559 Fatal exception in thread Thread[Hint >>>>> delivery,5,main] >>>> >>>> java.lang.OutOfMemoryError: Java heap space >>>> >>>> at java.util.Arrays.copyOf(Arrays.java:2772) >>>> >>>> at java.util.Arrays.copyOf(Arrays.java:2746) >>>> >>>> at java.util.ArrayList.ensureCapacity(ArrayList.java:187) >>>> >>>> at java.util.ArrayList.add(ArrayList.java:378) >>>> >>>> at >>>>> java.util.concurrent.ConcurrentSkipListMap.toList(ConcurrentSkipListM= ap.java:2341) >>>> >>>> at >>>>> java.util.concurrent.ConcurrentSkipListMap$Values.toArray(ConcurrentS= kipListMap.java:2445) >>>> >>>> at >>>>> org.apache.cassandra.db.Memtable.getSliceIterator(Memtable.java:207) >>>> >>>> at >>>>> org.apache.cassandra.db.filter.SliceQueryFilter.getMemColumnIterator(= SliceQueryFilter.java:58) >>>> >>>> at >>>>> org.apache.cassandra.db.filter.QueryFilter.getMemColumnIterator(Query= Filter.java:53) >>>> >>>> at >>>>> org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFa= milyStore.java:816) >>>> >>>> at >>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil= yStore.java:750) >>>> >>>> at >>>>> org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamil= yStore.java:719) >>>> >>>> at >>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHa= ndOffManager.java:175) >>>> >>>> at >>>>> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOff= Manager.java:80) >>>> >>>> at >>>>> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHand= OffManager.java:100) >>>> >>>> at >>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:3= 0) >>>> >>>> at java.lang.Thread.run(Thread.java:636) >>>> >>>> INFO 21:01:56,123 GC for ConcurrentMarkSweep: 2115 ms, 893240 reclaim= ed >>>>> leaving 1093862712 used; max is 1211826176 >>>> >>>> >>>> and >>>> >>>> ERROR [Hint delivery] 2010-04-19 21:57:07,089 CassandraDaemon.java (li= ne >>>>> 78) Fatal exception in thread Thread[Hint delivery,5,main] >>>> >>>> java.lang.RuntimeException: java.lang.RuntimeException: >>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: = Java >>>>> heap space >>>> >>>> at >>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:3= 4) >>>> >>>> at java.lang.Thread.run(Thread.java:636) >>>> >>>> Caused by: java.lang.RuntimeException: >>>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: = Java >>>>> heap space >>>> >>>> at >>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHa= ndOffManager.java:209) >>>> >>>> at >>>>> org.apache.cassandra.db.HintedHandOffManager.access$000(HintedHandOff= Manager.java:80) >>>> >>>> at >>>>> org.apache.cassandra.db.HintedHandOffManager$1.runMayThrow(HintedHand= OffManager.java:100) >>>> >>>> at >>>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:3= 0) >>>> >>>> ... 1 more >>>> >>>> Caused by: java.util.concurrent.ExecutionException: >>>>> java.lang.OutOfMemoryError: Java heap space >>>> >>>> at >>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) >>>> >>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111) >>>> >>>> at >>>>> org.apache.cassandra.db.HintedHandOffManager.deliverAllHints(HintedHa= ndOffManager.java:205) >>>> >>>> ... 4 more >>>> >>>> Caused by: java.lang.OutOfMemoryError: Java heap space >>>> >>>> ERROR [MESSAGE-DESERIALIZER-POOL:1] 2010-04-19 21:57:07,089 >>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuret= ask >>>> >>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: >>>>> Java heap space >>>> >>>> at >>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) >>>> >>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111) >>>> >>>> at >>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExe= cute(DebuggableThreadPoolExecutor.java:86) >>>> >>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1118) >>>> >>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:603) >>>> >>>> at java.lang.Thread.run(Thread.java:636) >>>> >>>> Caused by: java.lang.OutOfMemoryError: Java heap space >>>> >>>> ERROR [COMPACTION-POOL:1] 2010-04-19 21:57:07,089 >>>>> DebuggableThreadPoolExecutor.java (line 94) Error in executor futuret= ask >>>> >>>> java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: >>>>> Java heap space >>>> >>>> at >>>>> java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:252) >>>> >>>> at java.util.concurrent.FutureTask.get(FutureTask.java:111) >>>> >>>> at >>>>> org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.afterExe= cute(DebuggableThreadPoolExecutor.java:86) >>>> >>>> at >>>>> org.apache.cassandra.db.CompactionManager$CompactionExecutor.afterExe= cute(CompactionManager.java:582) >>>> >>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.= java:1118) >>>> >>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor= .java:603) >>>> >>>> at java.lang.Thread.run(Thread.java:636) >>>> >>>> Caused by: java.lang.OutOfMemoryError: Java heap space >>>> >>>> ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:56:29,572 CassandraDaemon.jav= a >>>>> (line 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main] >>>> >>>> java.lang.OutOfMemoryError: Java heap space >>>> >>>> at java.util.HashMap.(HashMap.java:226) at >>>>> org.apache.cassandra.utils.ExpiringMap$CacheMonitor.run(ExpiringMap.j= ava:76) >>>>> at java.util.TimerThread.mainLoop(Timer.java:534) at >>>>> java.util.TimerThread.run(Timer.java:484) >>>> >>>> >>>> >>>> >>> >>> >> > --00163630e9a57794d00484a18704 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable -Xmx1G is too small.
In my cluster, 8GB ram on each node, and I grant 6G= B to cassandra.

Please see my test @ http://www.slideshare.net/schubertzhang/= presentations

=E5=B9=BB=E7=81=AF=E7=89=87 5=
=E2=80=93Memory, GC..., always to be the bottle= neck and big issue o= f java-based infrastructure software!

References:
=E2=80=93http://wiki.a= pache.org/cassandra/FAQ#slows_down_after_lotso_inserts=20
=E2=80=93https://issues.apache.org/jir= a/browse/CASSANDRA-896=C2=A0 (LinkedBlockingQueue issue, fixed in jdk-6u19)


In fact, always when I using java-based infrastructure software, su= ch as Cassandra, Hadoop, HBase, etc, I am also pained about such memory/GC = issue finally.

Then, we should provide higher harware with more RAM= (such as 32GB~64GB), more CPU cores (such as 8~16). And we still cannot co= ntrol the Out-Of-Memory-Error.

I am thinking, maybe it is not right to leave the job of memory control= to JVM.

I have a long experience in telecom and embedded software i= n past ten years, where need robust programs and small RAM. I want to discu= ss following ideas with the community:

1. Manage the memory by ourselves: allocate objects/resource (memory) a= t initiating phase, and assign instances at runtime.
2. Reject the reque= st when be short of resource, instead of throws OOME and exit (crash).

3. I know, it is not easy in java program.

Schubert

On Tue, Apr 20, 2010 at 9:40 AM, Ken Sandney <blueflycn@gmail.com> wrote:
here is my JVM op= tions, by default, I didn't modify them, from cassandra.in.sh

# Arguments to pass to the JVM
JVM_OPTS=3D" \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-ea \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-Xms128M \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-Xmx1G \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-XX:TargetSurvivorRatio=3D90 \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-XX:+AggressiveOpts \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-XX:+UseParNewGC \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-XX:+UseConcMarkSweepGC \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-XX:+CMSParallelRemarkEnabled \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-XX:+HeapDumpOnOutOfMemoryError \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-XX:SurvivorRatio=3D128 \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-XX:MaxTenuringThreshold=3D0 \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-Dcom.sun.management.jmxremote.port=3D8080= \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-Dcom.sun.management.jmxremote.ssl=3Dfalse= \
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0-Dcom.sun.management.jmxremote.authenticat= e=3Dfalse"

and my box is normal pc wit= h 2GB ram, Intel E3200 =C2=A0@ 2.40GHz. By the way, I am using the latest S= un JDK

On Tue, Apr 20, 2010 at 9:33 AM, Schubert Zhang <zsongbo@gmail.com>= wrote:
Seems you should configure larger jvm-heap.


On Tue, Apr 20, 2010 at 9:32 AM, Schubert Zhang <z= songbo@gmail.com> wrote:
Please also post = your jvm-heap and GC options, i.e. the seting in cassandra.in.sh
And what about you node hardware?

On Tue, Apr 20, 2010 at 9:22 AM, Ken Sandney <blueflycn= @gmail.com> wrote:
Hi
I am doing a insert test with 9 nodes, the command:
stress.py -n 1000000000 -t 1000 -c 10 -o insert -i 5 -d 10.0.0.1,10.0.0.2..= ...
and =C2=A05 of the 9 nodes were cashed, only about 6&#= 39;500'000 rows were inserted
I checked out the system.log an= d seems the reason are 'out of memory'. I don't if this had som= ething to do with my settings.=C2=A0
Any idea about this?
Thank you, and the following are the er= rors from system.log
=C2=A0
ERROR [CACHETABLE-TIMER-1] 2010-04-19 20:43:14,013 CassandraDaemon.java (li= ne 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main]
java.lang.OutOfMemoryError: Java heap space
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.utils.ExpiringMap$= CacheMonitor.run(ExpiringMap.java:76)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.TimerThread.mainLoop(Timer.ja= va:512)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.TimerThread.run(Timer.java:46= 2)
ERROR [ROW-MUTATION-STAGE:9] 2010-04-19 20:43:27,932 CassandraDaemon.java (= line 78) Fatal exception in thread Thread[ROW-MUTATION-STAGE:9,5,main]
java.lang.OutOfMemoryError: Java heap space
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ConcurrentSkipList= Map.doPut(ConcurrentSkipListMap.java:893)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ConcurrentSkipList= Map.putIfAbsent(ConcurrentSkipListMap.java:1893)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.ColumnFamily.ad= dColumn(ColumnFamily.java:192)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.ColumnFamilySer= ializer.deserializeColumns(ColumnFamilySerializer.java:118)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.ColumnFamilySer= ializer.deserialize(ColumnFamilySerializer.java:108)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.RowMutationSeri= alizer.defreezeTheMaps(RowMutation.java:359)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.RowMutationSeri= alizer.deserialize(RowMutation.java:369)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.RowMutationSeri= alizer.deserialize(RowMutation.java:322)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.RowMutationVerb= Handler.doVerb(RowMutationVerbHandler.java:45)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.net.MessageDeliver= yTask.run(MessageDeliveryTask.java:40)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ThreadPoolExecutor= $Worker.runTask(ThreadPoolExecutor.java:886)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ThreadPoolExecutor= $Worker.run(ThreadPoolExecutor.java:908)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.java:619)

and another

=C2=A0INFO [GC inspection] 2010-04-19 21:13:09,034 GCInspector.java (line 1= 10) GC for ConcurrentMarkSweep: 2016 ms, 1239096 reclaimed leaving 10942389= 44 used; max is 1211826176
ERROR [Thread-14] 2010-04-19 21:23:18,508 CassandraDaemon.java (line 78) Fa= tal exception in thread Thread[Thread-14,5,main]
java.lang.OutOfMemoryError: Java heap space
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.Util.releaseTemporaryDirectB= uffer(Util.java:67)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.IOUtil.read(IOUtil.java:212)=
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.SocketChannelImpl.read(Socke= tChannelImpl.java:236)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.SocketAdaptor$SocketInputStr= eam.read(SocketAdaptor.java:176)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.ChannelInputStream.read(Chan= nelInputStream.java:86)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.io.InputStream.read(InputStream.ja= va:85)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at sun.nio.ch.ChannelInputStream.read(Chan= nelInputStream.java:64)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.io.DataInputStream.readInt(DataInp= utStream.java:370)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.net.IncomingTcpCon= nection.run(IncomingTcpConnection.java:70)
ERROR [COMPACTION-POOL:1] 2010-04-19 21:23:18,514 DebuggableThreadPoolExecu= tor.java (line 94) Error in executor futuretask
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java h= eap space
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.FutureTask$Sync.in= nerGet(FutureTask.java:222)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.FutureTask.get(Fut= ureTask.java:83)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.concurrent.Debugga= bleThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.CompactionManag= er$CompactionExecutor.afterExecute(CompactionManager.java:582)
=
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ThreadPoolExecutor= $Worker.runTask(ThreadPoolExecutor.java:888)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ThreadPoolExecutor= $Worker.run(ThreadPoolExecutor.java:908)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.java:619)
Caused by: java.lang.OutOfMemoryError: Java heap space
=C2=A0INFO [FLUSH-WRITER-POOL:1] 2010-04-19 21:23:25,600 Memtable.java (lin= e 162) Completed flushing /m/cassandra/data/Keyspace1/Standard1-623-Data.db=
ERROR [Thread-13] 2010-04-19 21:23:18,514 CassandraDaemon.java (line 78) Fa= tal exception in thread Thread[Thread-13,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [Thread-15] 2010-04-19 21:23:18,514 CassandraDaemon.java (line 78) Fa= tal exception in thread Thread[Thread-15,5,main]
java.lang.OutOfMemoryError: Java heap space
ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:23:18,514 CassandraDaemon.java (li= ne 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main]
java.lang.OutOfMemoryError: Java heap space
=C2=A0
and

=C2=A0INFO 21:00:31,319 GC for ConcurrentMarkSweep: 1417 ms, 206216 reclaim= ed leaving 1094527752 used; max is 1211826176
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid28670.hprof ...
=C2=A0INFO 21:01:23,882 GC for ConcurrentMarkSweep: 2100 ms, 734008 reclaim= ed leaving 1093996648 used; max is 1211826176
Heap dump file created [1095841554 bytes in 12.960 secs]
=C2=A0INFO 21:01:45,082 GC for ConcurrentMarkSweep: 2089 ms, 769968 reclaim= ed leaving 1093960776 used; max is 1211826176
ERROR 21:01:49,559 Fatal exception in thread Thread[Hint delivery,5,main]
java.lang.OutOfMemoryError: Java heap space
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.Arrays.copyOf(Arrays.java:277= 2)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.Arrays.copyOf(Arrays.java:274= 6)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.ArrayList.ensureCapacity(Arra= yList.java:187)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.ArrayList.add(ArrayList.java:= 378)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ConcurrentSkipList= Map.toList(ConcurrentSkipListMap.java:2341)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ConcurrentSkipList= Map$Values.toArray(ConcurrentSkipListMap.java:2445)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.Memtable.getSli= ceIterator(Memtable.java:207)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.filter.SliceQue= ryFilter.getMemColumnIterator(SliceQueryFilter.java:58)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.filter.QueryFil= ter.getMemColumnIterator(QueryFilter.java:53)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.ColumnFamilySto= re.getTopLevelColumns(ColumnFamilyStore.java:816)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.ColumnFamilySto= re.getColumnFamily(ColumnFamilyStore.java:750)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.ColumnFamilySto= re.getColumnFamily(ColumnFamilyStore.java:719)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.HintedHandOffMa= nager.deliverAllHints(HintedHandOffManager.java:175)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.HintedHandOffMa= nager.access$000(HintedHandOffManager.java:80)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.HintedHandOffMa= nager$1.runMayThrow(HintedHandOffManager.java:100)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.utils.WrappedRunna= ble.run(WrappedRunnable.java:30)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.java:636)
=C2=A0INFO 21:01:56,123 GC for ConcurrentMarkSweep: 2115 ms, 893240 reclaim= ed leaving 1093862712 used; max is 1211826176

and

ERROR [Hint delivery] 2010-04-19 21:57:07,089 CassandraDaemon.java (line 78= ) Fatal exception in thread Thread[Hint delivery,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.util.concurren= t.ExecutionException: java.lang.OutOfMemoryError: Java heap space
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.utils.WrappedRunna= ble.run(WrappedRunnable.java:34)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.RuntimeException: java.util.concurrent.ExecutionExcept= ion: java.lang.OutOfMemoryError: Java heap space
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.HintedHandOffMa= nager.deliverAllHints(HintedHandOffManager.java:209)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.HintedHandOffMa= nager.access$000(HintedHandOffManager.java:80)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.HintedHandOffMa= nager$1.runMayThrow(HintedHandOffManager.java:100)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.utils.WrappedRunna= ble.run(WrappedRunnable.java:30)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0... 1 more
Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryEr= ror: Java heap space
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.FutureTask$Sync.in= nerGet(FutureTask.java:252)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.FutureTask.get(Fut= ureTask.java:111)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.HintedHandOffMa= nager.deliverAllHints(HintedHandOffManager.java:205)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0... 4 more
Caused by: java.lang.OutOfMemoryError: Java heap space
ERROR [MESSAGE-DESERIALIZER-POOL:1] 2010-04-19 21:57:07,089 DebuggableThrea= dPoolExecutor.java (line 94) Error in executor futuretask java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java h= eap space
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.FutureTask$Sync.in= nerGet(FutureTask.java:252)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.FutureTask.get(Fut= ureTask.java:111)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.concurrent.Debugga= bleThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ThreadPoolExecutor= .runWorker(ThreadPoolExecutor.java:1118)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ThreadPoolExecutor= $Worker.run(ThreadPoolExecutor.java:603)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.OutOfMemoryError: Java heap space
ERROR [COMPACTION-POOL:1] 2010-04-19 21:57:07,089 DebuggableThreadPoolExecu= tor.java (line 94) Error in executor futuretask
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java h= eap space
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.FutureTask$Sync.in= nerGet(FutureTask.java:252)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.FutureTask.get(Fut= ureTask.java:111)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.concurrent.Debugga= bleThreadPoolExecutor.afterExecute(DebuggableThreadPoolExecutor.java:86)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.db.CompactionManag= er$CompactionExecutor.afterExecute(CompactionManager.java:582)
=
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ThreadPoolExecutor= .runWorker(ThreadPoolExecutor.java:1118)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.concurrent.ThreadPoolExecutor= $Worker.run(ThreadPoolExecutor.java:603)
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.lang.Thread.run(Thread.java:636)
Caused by: java.lang.OutOfMemoryError: Java heap space
ERROR [CACHETABLE-TIMER-1] 2010-04-19 21:56:29,572 CassandraDaemon.java (li= ne 78) Fatal exception in thread Thread[CACHETABLE-TIMER-1,5,main]
java.lang.OutOfMemoryError: Java heap space
=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0at java.util.HashMap.<init>(HashMap.= java:226) =C2=A0 =C2=A0 =C2=A0 =C2=A0at org.apache.cassandra.utils.Expiring= Map$CacheMonitor.run(ExpiringMap.java:76) =C2=A0 =C2=A0 =C2=A0 =C2=A0at jav= a.util.TimerThread.mainLoop(Timer.java:534) =C2=A0 =C2=A0 =C2=A0 =C2=A0at j= ava.util.TimerThread.run(Timer.java:484)

=C2=A0




--00163630e9a57794d00484a18704--