Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 13197 invoked from network); 4 Aug 2010 01:56:52 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 4 Aug 2010 01:56:52 -0000 Received: (qmail 47141 invoked by uid 500); 4 Aug 2010 01:56:50 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 47090 invoked by uid 500); 4 Aug 2010 01:56:50 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 47081 invoked by uid 99); 4 Aug 2010 01:56:50 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 01:56:50 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=FREEMAIL_FROM,HTML_MESSAGE,NORMAL_HTTP_TO_IP,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of prozect.mail@gmail.com designates 209.85.161.44 as permitted sender) Received: from [209.85.161.44] (HELO mail-fx0-f44.google.com) (209.85.161.44) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Aug 2010 01:56:45 +0000 Received: by fxm16 with SMTP id 16so1464747fxm.31 for ; Tue, 03 Aug 2010 18:56:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:in-reply-to :references:date:message-id:subject:from:to:content-type; bh=Bf9A/7VAm3EuubQbMMqXH3oqwmwlbzzgmp5xubnYl1A=; b=Hrw1r5izG7dUt0T11MhO/zKubBJwuCcpgK5UziCTU0NjIwJMwjpaH68XHsFR1DkV/4 uovLJQmpSBTxGCJ7xhenCjL61Wd8uSj229FtQNaZPUZ31Ue+GUhwH23kU87ZAt37Wko1 9kl9auYEq20hIYsevLAhlkGZPKNvlFMPwdWvU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=PLv3nuypvpSKhJtZRGuZMnFArID/ZCY1pzuB9g+ItWcjwmOBTYOOkOu6qeIqQqOu0b 9Hj2V9iOTkfqXpNEzw7I+ivSVxrQljerEEiKQUkUPsRHGriCklN5n1uAdg3DlfKNOG7u tDMMhv2UM2UwO0GoFp+OViEbBUWz+YzFk7wq4= MIME-Version: 1.0 Received: by 10.239.132.136 with SMTP id 8mr251448hbr.60.1280886983566; Tue, 03 Aug 2010 18:56:23 -0700 (PDT) Received: by 10.239.163.74 with HTTP; Tue, 3 Aug 2010 18:56:23 -0700 (PDT) In-Reply-To: References: Date: Wed, 4 Aug 2010 10:56:23 +0900 Message-ID: Subject: Re: Two questions : Server crash during compaction and UnavailableException From: Ilun Ahn To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001485f5b188a72830048cf5bd0e --001485f5b188a72830048cf5bd0e Content-Type: text/plain; charset=ISO-8859-1 You're right... I missed posting crash log. I was too busy and under press of business at that time. Please understand. These are head and tail of the JVM crash log when it stopped : --------------- T H R E A D --------------- Current thread (0x0000002ca9903400): JavaThread "COMPACTION-POOL:1" [_thread_in_vm, id=10768] siginfo:si_signo=11, si_errno=0, si_code=128, si_addr=0x0000000000000000 Registers: RAX=0x0000002cad287d10, RBX=0x0000002ca9903400, RCX=0x0000002cad2880f8, RDX=0x0000002cad287d20 RSP=0x0000000042149160, RBP=0x0000000042149170, RSI=0x0000002b0b7e40c8, RDI=0x0100048c9b42085b R8 =0x0000002cac795910, R9 =0x0000000000002a10, R10=0x0000002a95f5d200, R11=0x0000002cac795910 R12=0x0000002b20d2cb90, R13=0x00000000421491c0, R14=0x0000000042149268, R15=0x0000002ca9903400 RIP=0x0000002a95c05727, EFL=0x0000000000010206, CSGSFS=0x0000000000000002, ERR=0x0000000000000000 TRAPNO=0x000000000000000d Top of Stack: (sp=0x0000000042149160) 0x0000000042149160: 0000002ca9903400 0000002b20d2cb90 0x0000000042149170: 0000000042149180 0000002a95ab6fcd 0x0000000042149180: 00000000421491f0 0000002a95957896 0x0000000042149190: 0000002cac795910 0000002cad287d10 0x00000000421491a0: 0000002cad287d20 0000002cad2880f8 0x00000000421491b0: 0000002ca9903400 0000000000000013 0x00000000421491c0: 0000002ca9903400 0000002ca9903400 0x00000000421491d0: 00000000421491e0 0000002b0b7e4099 0x00000000421491e0: 0000002b20d2cb90 0000002c99d6850e 0x00000000421491f0: 0000000042149248 0000002a96512bdd 0x0000000042149200: 0000002a96512b9b 0000000042149208 0x0000000042149210: 0000002c99d6850e 0000000042149268 0x0000000042149220: 0000002c9a021658 0000002c9a3e6c08 0x0000000042149230: 0000002c99d68540 0000000000000000 0x0000000042149240: 0000000042149258 00000000421492b0 0x0000000042149250: 0000002a96509528 0000000000000000 0x0000000042149260: 0000000000000000 0000000000000000 0x0000000042149270: 0000000042149270 0000002c9a1f7159 0x0000000042149280: 00000000421492c8 0000002c9a1f7518 0x0000000042149290: 0000002c9a3e7200 0000002c9a1f7168 0x00000000421492a0: 0000000042149258 00000000421492c0 0x00000000421492b0: 0000000042149310 0000002a96509528 0x00000000421492c0: 0000000000000000 0000000000000000 0x00000000421492d0: 00000000421492d0 0000002c9a0966c5 0x00000000421492e0: 0000000042149348 0000002c9a09bf90 0x00000000421492f0: 0000002c9a3e6e48 0000002c9a096718 0x0000000042149300: 00000000421492c0 0000000042149350 0x0000000042149310: 0000002a9e9fd1a8 0000002a96815d80 0x0000000042149320: 0000000000000000 0000000000000000 0x0000000042149330: 0000000000000000 0000000000000000 0x0000000042149340: 0000000000000000 0000000000000000 0x0000000042149350: 0000002ca9903400 0000002b20d2cbc0 Instructions: (pc=0x0000002a95c05727) 0x0000002a95c05717: 89 f0 eb ea 90 66 66 66 90 55 48 89 e5 41 54 53 0x0000002a95c05727: 0f b7 57 10 48 89 fb 44 8d 62 01 49 63 fc e8 c6 Stack: [0x000000004204a000,0x000000004214b000), sp=0x0000000042149160, free space=1020k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x592727] V [libjvm.so+0x443fcd] V [libjvm.so+0x2e4896] j org.apache.cassandra.db.marshal.BytesType.compare(Ljava/lang/Object;Ljava/lang/Object;)I+6 j java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(Ljava/lang/Object;)I+9 j java.util.concurrent.ConcurrentSkipListMap.findPredecessor(Ljava/lang/Comparable;)Ljava/util/concurrent/ConcurrentSkipListMap$Node;+69 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j org.apache.cassandra.db.marshal.BytesType.compare(Ljava/lang/Object;Ljava/lang/Object;)I+6 j java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.compareTo(Ljava/lang/Object;)I+9 j java.util.concurrent.ConcurrentSkipListMap.findPredecessor(Ljava/lang/Comparable;)Ljava/util/concurrent/ConcurrentSkipListMap$Node;+69 J java.util.concurrent.ConcurrentSkipListMap.doPut(Ljava/lang/Object;Ljava/lang/Object;Z)Ljava/lang/Object; J org.apache.cassandra.db.ColumnFamily.addColumn(Lorg/apache/cassandra/db/IColumn;)V J org.apache.cassandra.db.ColumnFamily.addAll(Lorg/apache/cassandra/db/ColumnFamily;)V J org.apache.cassandra.io.CompactionIterator.getReduced()Lorg/apache/cassandra/io/CompactionIterator$CompactedRow; J org.apache.cassandra.io.CompactionIterator.getReduced()Ljava/lang/Object; J org.apache.cassandra.utils.ReducingIterator.computeNext()Ljava/lang/Object; J com.google.common.collect.AbstractIterator.hasNext()Z J org.apache.cassandra.db.CompactionManager.doAntiCompaction(Lorg/apache/cassandra/db/ColumnFamilyStore;Ljava/util/Collection;Ljava/util/Collection;Ljava/net/InetAddress;)Ljava/util/List; j org.apache.cassandra.db.CompactionManager.doCleanupCompaction(Lorg/apache/cassandra/db/ColumnFamilyStore;)V+22 j org.apache.cassandra.db.CompactionManager.access$400(Lorg/apache/cassandra/db/CompactionManager;Lorg/apache/cassandra/db/ColumnFamilyStore;)V+2 j org.apache.cassandra.db.CompactionManager$2.call()Ljava/lang/Object;+8 J java.util.concurrent.FutureTask$Sync.innerRun()V j java.util.concurrent.FutureTask.run()V+4 j java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Ljava/lang/Runnable;)V+59 j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+28 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0x0000002cad6ff800 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=16294] 0x0000002cad21b800 JavaThread "Thread-11" [_thread_in_native, id=13570] 0x0000002cad27dc00 JavaThread "Thread-10" [_thread_in_native, id=11221] 0x0000002caceccc00 JavaThread "WRITE-/10.25.32.37" [_thread_blocked, id=10830] 0x0000002cad33c000 JavaThread "HINTED-HANDOFF-POOL:1" [_thread_blocked, id=10829] 0x0000002cacd55c00 JavaThread "WRITE-/10.25.32.36" [_thread_blocked, id=10828] 0x0000002cacd54c00 JavaThread "WRITE-/10.25.32.34" [_thread_blocked, id=10827] 0x0000002cad2b5400 JavaThread "LOAD-BALANCER-STAGE:1" [_thread_blocked, id=10826] 0x0000002cad2b3c00 JavaThread "AE-SERVICE-STAGE:1" [_thread_blocked, id=10825] 0x0000002cad2b2800 JavaThread "GMFD:1" [_thread_blocked, id=10824] 0x0000002cad2b1000 JavaThread "STREAM-STAGE:1" [_thread_blocked, id=10823] 0x0000002cad2afc00 JavaThread "RESPONSE-STAGE:4" [_thread_blocked, id=10822] 0x0000002cad2ae800 JavaThread "RESPONSE-STAGE:3" [_thread_blocked, id=10821] 0x0000002cad2ad000 JavaThread "RESPONSE-STAGE:2" [_thread_blocked, id=10820] 0x0000002cad2abc00 JavaThread "RESPONSE-STAGE:1" [_thread_blocked, id=10819] 0x0000002cad2aa800 JavaThread "ROW-READ-STAGE:8" [_thread_blocked, id=10818] 0x0000002cad2a9c00 JavaThread "ROW-READ-STAGE:7" [_thread_blocked, id=10817] 0x0000002cad2a8400 JavaThread "ROW-READ-STAGE:6" [_thread_blocked, id=10816] 0x0000002cad2a6c00 JavaThread "ROW-READ-STAGE:5" [_thread_blocked, id=10815] 0x0000002cad2a5800 JavaThread "ROW-READ-STAGE:4" [_thread_blocked, id=10814] 0x0000002cacf82400 JavaThread "ROW-READ-STAGE:3" [_thread_blocked, id=10813] 0x0000002cacf80c00 JavaThread "ROW-READ-STAGE:2" [_thread_blocked, id=10812] 0x0000002cacf7f800 JavaThread "ROW-READ-STAGE:1" [_thread_blocked, id=10811] 0x0000002cacf7e400 JavaThread "ROW-MUTATION-STAGE:32" [_thread_blocked, id=10810] 0x0000002cacf7cc00 JavaThread "ROW-MUTATION-STAGE:31" [_thread_blocked, id=10809] 0x0000002cacf7b800 JavaThread "ROW-MUTATION-STAGE:30" [_thread_blocked, id=10808] 0x0000002cacf7a000 JavaThread "ROW-MUTATION-STAGE:29" [_thread_blocked, id=10807] 0x0000002cacf78c00 JavaThread "ROW-MUTATION-STAGE:28" [_thread_blocked, id=10806] 0x0000002cacf77800 JavaThread "ROW-MUTATION-STAGE:27" [_thread_blocked, id=10805] 0x0000002cacf76000 JavaThread "ROW-MUTATION-STAGE:26" [_thread_blocked, id=10804] 0x0000002cacf74c00 JavaThread "ROW-MUTATION-STAGE:25" [_thread_blocked, id=10803] 0x0000002cacf73400 JavaThread "ROW-MUTATION-STAGE:24" [_thread_blocked, id=10802] 0x0000002cacf72000 JavaThread "ROW-MUTATION-STAGE:23" [_thread_blocked, id=10801] 0x0000002cacf70c00 JavaThread "ROW-MUTATION-STAGE:22" [_thread_blocked, id=10800] 0x0000002cacf6f400 JavaThread "ROW-MUTATION-STAGE:21" [_thread_blocked, id=10799] 0x0000002cacf6e000 JavaThread "ROW-MUTATION-STAGE:20" [_thread_blocked, id=10798] 0x0000002cacf6c800 JavaThread "ROW-MUTATION-STAGE:19" [_thread_blocked, id=10797] 0x0000002cacf6b400 JavaThread "ROW-MUTATION-STAGE:18" [_thread_blocked, id=10796] 0x0000002cacf6a000 JavaThread "ROW-MUTATION-STAGE:17" [_thread_blocked, id=10795] 0x0000002cacf68800 JavaThread "ROW-MUTATION-STAGE:16" [_thread_blocked, id=10794] 0x0000002cacf67400 JavaThread "ROW-MUTATION-STAGE:15" [_thread_blocked, id=10793] 0x0000002cacf65c00 JavaThread "ROW-MUTATION-STAGE:14" [_thread_blocked, id=10792] 0x0000002cacf64800 JavaThread "ROW-MUTATION-STAGE:13" [_thread_blocked, id=10791] 0x0000002cacf63400 JavaThread "ROW-MUTATION-STAGE:12" [_thread_blocked, id=10790] 0x0000002cacf61c00 JavaThread "ROW-MUTATION-STAGE:11" [_thread_blocked, id=10789] 0x0000002cacf60800 JavaThread "ROW-MUTATION-STAGE:10" [_thread_blocked, id=10788] 0x0000002cacf5f000 JavaThread "ROW-MUTATION-STAGE:9" [_thread_blocked, id=10787] 0x0000002cacf5bc00 JavaThread "ROW-MUTATION-STAGE:8" [_thread_blocked, id=10786] 0x0000002cacf5a400 JavaThread "ROW-MUTATION-STAGE:7" [_thread_blocked, id=10785] 0x0000002cacf5d400 JavaThread "ROW-MUTATION-STAGE:6" [_thread_blocked, id=10784] 0x0000002cacf5c400 JavaThread "ROW-MUTATION-STAGE:5" [_thread_blocked, id=10783] 0x0000002cacf57800 JavaThread "ROW-MUTATION-STAGE:4" [_thread_blocked, id=10782] 0x0000002cad258400 JavaThread "ROW-MUTATION-STAGE:3" [_thread_blocked, id=10781] 0x0000002ca98f3400 JavaThread "ROW-MUTATION-STAGE:2" [_thread_blocked, id=10780] 0x0000002ca98f2400 JavaThread "ROW-MUTATION-STAGE:1" [_thread_blocked, id=10779] 0x0000002ca9902000 JavaThread "Thread-9" [_thread_in_native, id=10778] 0x0000002ca9901800 JavaThread "Thread-8" [_thread_in_native, id=10777] 0x0000002cad253000 JavaThread "Timer-1" [_thread_blocked, id=10776] 0x0000002cad25ec00 JavaThread "LB-TARGET:1" [_thread_blocked, id=10775] 0x0000002cad25d800 JavaThread "LB-OPERATIONS:1" [_thread_blocked, id=10774] 0x0000002cad25cc00 JavaThread "ACCEPT-krs1165.nhncorp.com/10.25.32.35" [_thread_in_native, id=10773] 0x0000002cad25bc00 JavaThread "Timer-0" [_thread_blocked, id=10772] 0x0000002cacf19400 JavaThread "GC inspection" [_thread_blocked, id=10771] 0x0000002cad28bc00 JavaThread "PERIODIC-COMMIT-LOG-SYNCER" [_thread_blocked, id=10770] 0x0000002cad2a4400 JavaThread "COMMIT-LOG-WRITER" [_thread_blocked, id=10769] =>0x0000002ca9903400 JavaThread "COMPACTION-POOL:1" [_thread_in_vm, id=10768] 0x0000002cad2a3400 JavaThread "SSTABLE-CLEANUP-TIMER" [_thread_blocked, id=10378] 0x0000002cace74c00 JavaThread "SSTABLE-DELETER" [_thread_blocked, id=10377] 0x0000002cad227000 JavaThread "MESSAGE-STREAMING-POOL:1" [_thread_blocked, id=10376] 0x0000002cad225c00 JavaThread "MESSAGE-DESERIALIZER-POOL:1" [_thread_blocked, id=10375] 0x0000002cad224800 JavaThread "CACHETABLE-TIMER-2" daemon [_thread_blocked, id=10374] 0x0000002cad224000 JavaThread "CACHETABLE-TIMER-1" daemon [_thread_blocked, id=10373] 0x0000002cacf55c00 JavaThread "CONSISTENCY-MANAGER:4" [_thread_blocked, id=10372] 0x0000002cacfa1800 JavaThread "CONSISTENCY-MANAGER:3" [_thread_blocked, id=10371] 0x0000002cacf9cc00 JavaThread "CONSISTENCY-MANAGER:2" [_thread_blocked, id=10370] 0x0000002cacfa2800 JavaThread "CONSISTENCY-MANAGER:1" [_thread_blocked, id=10369] 0x0000002cacf9b000 JavaThread "MEMTABLE-POST-FLUSHER:1" [_thread_blocked, id=10368] 0x0000002cad230800 JavaThread "FLUSH-WRITER-POOL:1" [_thread_blocked, id=10367] 0x0000002cad237c00 JavaThread "FLUSH-SORTER-POOL:1" [_thread_blocked, id=10366] 0x0000002cad230400 JavaThread "FLUSH-TIMER" [_thread_blocked, id=10365] 0x0000002cacda6800 JavaThread "RMI TCP Accept-8080" daemon [_thread_in_native, id=10363] 0x0000002cacd48800 JavaThread "RMI TCP Accept-0" daemon [_thread_in_native, id=10362] 0x0000002ca9893400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=10361] 0x0000002ca9891400 JavaThread "CompilerThread1" daemon [_thread_blocked, id=10360] 0x0000002ca988f800 JavaThread "CompilerThread0" daemon [_thread_blocked, id=10359] 0x0000002ca988e000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=10358] 0x0000002ca988c800 JavaThread "Surrogate Locker Thread (CMS)" daemon [_thread_blocked, id=10357] 0x0000002ca9869c00 JavaThread "Finalizer" daemon [_thread_blocked, id=10356] 0x0000002ca9869000 JavaThread "Reference Handler" daemon [_thread_blocked, id=10355] 0x0000000040112800 JavaThread "main" [_thread_in_native, id=10348] Other Threads: 0x0000002ca9864400 VMThread [id=10354] 0x0000002cacd8d800 WatcherThread [id=10364] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap par new generation total 76672K, used 55139K [0x0000002a995d0000, 0x0000002a9e900000, 0x0000002a9e900000) eden space 68160K, 68% used [0x0000002a995d0000, 0x0000002a9c359758, 0x0000002a9d860000) from space 8512K, 99% used [0x0000002a9e0b0000, 0x0000002a9e8ff820, 0x0000002a9e900000) to space 8512K, 0% used [0x0000002a9d860000, 0x0000002a9d860000, 0x0000002a9e0b0000) concurrent mark-sweep generation total 5810916K, used 4774431K [0x0000002a9e900000, 0x0000002c013b9000, 0x0000002c995d0000) concurrent-mark-sweep perm gen total 26096K, used 15598K [0x0000002c995d0000, 0x0000002c9af4c000, 0x0000002c9e9d0000) ....(the rest omitted) --------------- S Y S T E M --------------- OS:CentOS release 4.5 (Final) uname:Linux 2.6.9-55.0.12.EL.nhnsmp #1 SMP Tue Nov 6 18:38:43 KST 2007 x86_64 libc:glibc 2.3.4 NPTL 2.3.4 rlimit: STACK 10240k, CORE 0k, NPROC 16383, NOFILE 32768, AS infinity load average:1.74 1.51 1.44 CPU:total 4 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3 Memory: 4k page, physical 16407020k(51816k free), swap 4192956k(4016836k free) vm_info: Java HotSpot(TM) 64-Bit Server VM (1.6.0_03-b05) for linux-amd64, built on Sep 24 2007 23:09:21 by "java_re" with gcc 3.2.2 (SuSE Linux) I set java option -Xmx8G and the others were unchanged from Cassandra 0.6.3 settings. I will try upgrading JVM first. Thanks for advice. :) 2010/8/3 Jonathan Ellis > If you have a crash log you should post at least the header rather > than playing 20 questions with us. But if it's not OOM then it's > likely to be a bug in the JVM, so upgrading is probably your best > option. > > On Tue, Aug 3, 2010 at 3:49 AM, Ilun Ahn wrote: > > No, I don't think the direct cause is out of heap space. It didn't left > any > > heap dump file with the option -XX:+HeapDumpOnOutOfMemoryError. > > My system.log for the last minute is as follows(many GC occurs): > > INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,254 > > HintedHandOffManager.java (line 153) Started hinted handoff for endPoint > > /10.25.32.36 > > INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,255 > > HintedHandOffManager.java (line 210) Finished hinted handoff of 0 rows to > > endpoint /10.25.32.36 > > INFO [GC inspection] 2010-08-02 20:34:01,919 GCInspector.java (line 110) > GC > > for ParNew: 269 ms, 11161808 reclaimed leaving 5068544312 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:34:14,985 GCInspector.java (line 110) > GC > > for ParNew: 208 ms, 12326736 reclaimed leaving 4044195008 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:34:15,986 GCInspector.java (line 110) > GC > > for ParNew: 208 ms, 12283112 reclaimed leaving 2005777224 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:35:45,834 GCInspector.java (line 110) > GC > > for ParNew: 229 ms, 13074080 reclaimed leaving 5374833480 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:35:46,836 GCInspector.java (line 110) > GC > > for ParNew: 203 ms, 12529824 reclaimed leaving 5321733432 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:37:01,624 GCInspector.java (line 110) > GC > > for ParNew: 206 ms, 11029656 reclaimed leaving 4473650352 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:38:19,064 GCInspector.java (line 110) > GC > > for ConcurrentMarkSweep: 4501 ms, 1057548400 reclaimed leaving 2461458096 > > used; max is 8719630336 > > INFO [GC inspection] 2010-08-02 20:41:20,446 GCInspector.java (line 110) > GC > > for ParNew: 218 ms, 15072720 reclaimed leaving 5345683640 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:41:23,453 GCInspector.java (line 110) > GC > > for ParNew: 234 ms, 16818048 reclaimed leaving 3937902088 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:42:15,229 GCInspector.java (line 110) > GC > > for ConcurrentMarkSweep: 8015 ms, 739534984 reclaimed leaving 3550138024 > > used; max is 8719630336 > > INFO [GC inspection] 2010-08-02 20:42:42,444 GCInspector.java (line 110) > GC > > for ParNew: 203 ms, 14218928 reclaimed leaving 4398967608 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:42:43,565 GCInspector.java (line 110) > GC > > for ParNew: 203 ms, 12274600 reclaimed leaving 1989854648 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:43:01,801 GCInspector.java (line 110) > GC > > for ParNew: 212 ms, 10183184 reclaimed leaving 2337034168 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:44:01,215 GCInspector.java (line 110) > GC > > for ParNew: 218 ms, 10402368 reclaimed leaving 4334140184 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:44:35,623 GCInspector.java (line 110) > GC > > for ConcurrentMarkSweep: 4424 ms, 3101007888 reclaimed leaving 2459621048 > > used; max is 8719630336 > > INFO [GC inspection] 2010-08-02 20:45:32,089 GCInspector.java (line 110) > GC > > for ParNew: 227 ms, 27109720 reclaimed leaving 5410486832 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:45:35,095 GCInspector.java (line 110) > GC > > for ParNew: 203 ms, 28235832 reclaimed leaving 3580093424 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:46:01,203 GCInspector.java (line 110) > GC > > for ParNew: 257 ms, 12257744 reclaimed leaving 3469012312 used; max is > > 8719630336 > > INFO [GC inspection] 2010-08-02 20:46:51,060 GCInspector.java (line 110) > GC > > for ParNew: 222 ms, 18473064 reclaimed leaving 5320004640 used; max is > > 8719630336 > > > > > > > > > > > > and the JRE crash log show that current thread is COMPACTION-POOL:1. > > > > > > > > ps. Sorry for unnecessary message. my mistake. > > > > > > > > > > 2010/8/3 Ilun Ahn > >> > >> > >> 2010/8/2 Peter Schuller > >>> > >>> > First, Cassandra suddenly dies during compaction. Java core dump says > >>> > that > >>> > the last thread run was "COMPACTION-POOL:1". > >>> > I suspect that my business logic could lead size of columns in a > column > >>> > family per a row to be greater than two gigabytes. (but i couldn't > >>> > confirm > >>> > it yet) > >>> > >>> Are you running out of memory (java heap)? If you're running cassandra > >>> with default options, it will be running with > >>> -XX:+HeapDumpOnOutOfMemoryError > >>> > >>> Have you checked the cassandra system.log for garbage collection > >>> messages? What is in the last minute or two of logs? > >>> > >>> -- > >>> / Peter Schuller > >> > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of Riptano, the source for professional Cassandra support > http://riptano.com > --001485f5b188a72830048cf5bd0e Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable
You're right... I missed posting crash log.
I was too b= usy and under press of business at that time. Please understand.
<= div>


These are head and tail of= the JVM crash log when it stopped :


--------------- T H R E A D ----------= -----

Current thread (0x0000002ca9903400): JavaThread "COMPACT= ION-POOL:1" [_thread_in_vm, id=3D10768]

siginfo:si_signo=3D11, = si_errno=3D0, si_code=3D128, si_addr=3D0x0000000000000000

Registers:
RAX=3D0x0000002cad287d10, RBX=3D0x0000002ca9903400, RCX= =3D0x0000002cad2880f8, RDX=3D0x0000002cad287d20
RSP=3D0x0000000042149160= , RBP=3D0x0000000042149170, RSI=3D0x0000002b0b7e40c8, RDI=3D0x0100048c9b420= 85b
R8 =3D0x0000002cac795910, R9 =3D0x0000000000002a10, R10=3D0x0000002a= 95f5d200, R11=3D0x0000002cac795910
R12=3D0x0000002b20d2cb90, R13=3D0x00000000421491c0, R14=3D0x000000004214926= 8, R15=3D0x0000002ca9903400
RIP=3D0x0000002a95c05727, EFL=3D0x0000000000= 010206, CSGSFS=3D0x0000000000000002, ERR=3D0x0000000000000000
TRAPNO= =3D0x000000000000000d

Top of Stack: (sp=3D0x0000000042149160)
0x0000000042149160: 000000= 2ca9903400 0000002b20d2cb90
0x0000000042149170: 0000000042149180 00000= 02a95ab6fcd
0x0000000042149180: 00000000421491f0 0000002a95957896
0x0000000042149190: 0000002cac795910 0000002cad287d10
0x00000000421491= a0: 0000002cad287d20 0000002cad2880f8
0x00000000421491b0: 0000002ca9= 903400 0000000000000013
0x00000000421491c0: 0000002ca9903400 0000002ca= 9903400
0x00000000421491d0: 00000000421491e0 0000002b0b7e4099
0x00000000421491= e0: 0000002b20d2cb90 0000002c99d6850e
0x00000000421491f0: 0000000042= 149248 0000002a96512bdd
0x0000000042149200: 0000002a96512b9b 000000004= 2149208
0x0000000042149210: 0000002c99d6850e 0000000042149268
0x00000000421492= 20: 0000002c9a021658 0000002c9a3e6c08
0x0000000042149230: 0000002c99= d68540 0000000000000000
0x0000000042149240: 0000000042149258 000000004= 21492b0
0x0000000042149250: 0000002a96509528 0000000000000000
0x00000000421492= 60: 0000000000000000 0000000000000000
0x0000000042149270: 0000000042= 149270 0000002c9a1f7159
0x0000000042149280: 00000000421492c8 0000002c9= a1f7518
0x0000000042149290: 0000002c9a3e7200 0000002c9a1f7168
0x00000000421492= a0: 0000000042149258 00000000421492c0
0x00000000421492b0: 0000000042= 149310 0000002a96509528
0x00000000421492c0: 0000000000000000 000000000= 0000000
0x00000000421492d0: 00000000421492d0 0000002c9a0966c5
0x00000000421492= e0: 0000000042149348 0000002c9a09bf90
0x00000000421492f0: 0000002c9a= 3e6e48 0000002c9a096718
0x0000000042149300: 00000000421492c0 000000004= 2149350
0x0000000042149310: 0000002a9e9fd1a8 0000002a96815d80
0x00000000421493= 20: 0000000000000000 0000000000000000
0x0000000042149330: 0000000000= 000000 0000000000000000
0x0000000042149340: 0000000000000000 000000000= 0000000
0x0000000042149350: 0000002ca9903400 0000002b20d2cbc0

I= nstructions: (pc=3D0x0000002a95c05727)
0x0000002a95c05717: 89 f0 eb ea= 90 66 66 66 90 55 48 89 e5 41 54 53
0x0000002a95c05727: 0f b7 57 10 4= 8 89 fb 44 8d 62 01 49 63 fc e8 c6

Stack: [0x000000004204a000,0x000000004214b000), sp=3D0x000000004214916= 0, free space=3D1020k
Native frames: (J=3Dcompiled Java code, j=3Dinter= preted, Vv=3DVM code, C=3Dnative code)
V [libjvm.so+0x592727]
V [li= bjvm.so+0x443fcd]
V [libjvm.so+0x2e4896]
j org.apache.cassandra.db.marshal.BytesType.com= pare(Ljava/lang/Object;Ljava/lang/Object;)I+6
j java.util.concurrent.Co= ncurrentSkipListMap$ComparableUsingComparator.compareTo(Ljava/lang/Object;)= I+9
j java.util.concurrent.ConcurrentSkipListMap.findPredecessor(Ljava/lang/Co= mparable;)Ljava/util/concurrent/ConcurrentSkipListMap$Node;+69

Java = frames: (J=3Dcompiled Java code, j=3Dinterpreted, Vv=3DVM code)
j org.a= pache.cassandra.db.marshal.BytesType.compare(Ljava/lang/Object;Ljava/lang/O= bject;)I+6
j java.util.concurrent.ConcurrentSkipListMap$ComparableUsingComparator.com= pareTo(Ljava/lang/Object;)I+9
j java.util.concurrent.ConcurrentSkipList= Map.findPredecessor(Ljava/lang/Comparable;)Ljava/util/concurrent/Concurrent= SkipListMap$Node;+69
J java.util.concurrent.ConcurrentSkipListMap.doPut(Ljava/lang/Object;Ljava= /lang/Object;Z)Ljava/lang/Object;
J org.apache.cassandra.db.ColumnFamil= y.addColumn(Lorg/apache/cassandra/db/IColumn;)V
J org.apache.cassandra.= db.ColumnFamily.addAll(Lorg/apache/cassandra/db/ColumnFamily;)V
J org.apache.cassandra.io.CompactionIterator.getReduced()Lorg/apache/cassa= ndra/io/CompactionIterator$CompactedRow;
J org.apache.cassandra.io.Comp= actionIterator.getReduced()Ljava/lang/Object;
J org.apache.cassandra.ut= ils.ReducingIterator.computeNext()Ljava/lang/Object;
J com.google.common.collect.AbstractIterator.hasNext()Z
J org.apache.c= assandra.db.CompactionManager.doAntiCompaction(Lorg/apache/cassandra/db/Col= umnFamilyStore;Ljava/util/Collection;Ljava/util/Collection;Ljava/net/InetAd= dress;)Ljava/util/List;
j org.apache.cassandra.db.CompactionManager.doCleanupCompaction(Lorg/apach= e/cassandra/db/ColumnFamilyStore;)V+22
j org.apache.cassandra.db.Compac= tionManager.access$400(Lorg/apache/cassandra/db/CompactionManager;Lorg/apac= he/cassandra/db/ColumnFamilyStore;)V+2
j org.apache.cassandra.db.CompactionManager$2.call()Ljava/lang/Object;+8J java.util.concurrent.FutureTask$Sync.innerRun()V
j java.util.concu= rrent.FutureTask.run()V+4
j java.util.concurrent.ThreadPoolExecutor$Wor= ker.runTask(Ljava/lang/Runnable;)V+59
j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+28
j java.lang= .Thread.run()V+11
v ~StubRoutines::call_stub

--------= ------- P R O C E S S ---------------

Java Threads: ( =3D> curr= ent thread )
0x0000002cad6ff800 JavaThread "RMI Scheduler(0)" daemon [_threa= d_blocked, id=3D16294]
0x0000002cad21b800 JavaThread "Thread-11&q= uot; [_thread_in_native, id=3D13570]
0x0000002cad27dc00 JavaThread &qu= ot;Thread-10" [_thread_in_native, id=3D11221]
0x0000002caceccc00 JavaThread "WRITE-/10.25.32.37" [_thread_blocked, id=3D10830]
0x0000002cad33c00= 0 JavaThread "HINTED-HANDOFF-POOL:1" [_thread_blocked, id=3D10829= ]
0x0000002cacd55c00 JavaThread "WRITE-/10.25.32.36" [_thread_blocked, id=3D10828]
0x0000002cacd54c0= 0 JavaThread "WRITE-/10.25.32.34&qu= ot; [_thread_blocked, id=3D10827]
0x0000002cad2b5400 JavaThread "LOAD-BALANCER-STAGE:1" [_thread_= blocked, id=3D10826]
0x0000002cad2b3c00 JavaThread "AE-SERVICE-ST= AGE:1" [_thread_blocked, id=3D10825]
0x0000002cad2b2800 JavaThrea= d "GMFD:1" [_thread_blocked, id=3D10824]
0x0000002cad2b1000 JavaThread "STREAM-STAGE:1" [_thread_blocked= , id=3D10823]
0x0000002cad2afc00 JavaThread "RESPONSE-STAGE:4&quo= t; [_thread_blocked, id=3D10822]
0x0000002cad2ae800 JavaThread "R= ESPONSE-STAGE:3" [_thread_blocked, id=3D10821]
0x0000002cad2ad000 JavaThread "RESPONSE-STAGE:2" [_thread_block= ed, id=3D10820]
0x0000002cad2abc00 JavaThread "RESPONSE-STAGE:1&q= uot; [_thread_blocked, id=3D10819]
0x0000002cad2aa800 JavaThread "= ;ROW-READ-STAGE:8" [_thread_blocked, id=3D10818]
0x0000002cad2a9c00 JavaThread "ROW-READ-STAGE:7" [_thread_block= ed, id=3D10817]
0x0000002cad2a8400 JavaThread "ROW-READ-STAGE:6&q= uot; [_thread_blocked, id=3D10816]
0x0000002cad2a6c00 JavaThread "= ;ROW-READ-STAGE:5" [_thread_blocked, id=3D10815]
0x0000002cad2a5800 JavaThread "ROW-READ-STAGE:4" [_thread_block= ed, id=3D10814]
0x0000002cacf82400 JavaThread "ROW-READ-STAGE:3&q= uot; [_thread_blocked, id=3D10813]
0x0000002cacf80c00 JavaThread "= ;ROW-READ-STAGE:2" [_thread_blocked, id=3D10812]
0x0000002cacf7f800 JavaThread "ROW-READ-STAGE:1" [_thread_block= ed, id=3D10811]
0x0000002cacf7e400 JavaThread "ROW-MUTATION-STAGE= :32" [_thread_blocked, id=3D10810]
0x0000002cacf7cc00 JavaThread = "ROW-MUTATION-STAGE:31" [_thread_blocked, id=3D10809]
0x0000002cacf7b800 JavaThread "ROW-MUTATION-STAGE:30" [_thread_= blocked, id=3D10808]
0x0000002cacf7a000 JavaThread "ROW-MUTATION-= STAGE:29" [_thread_blocked, id=3D10807]
0x0000002cacf78c00 JavaTh= read "ROW-MUTATION-STAGE:28" [_thread_blocked, id=3D10806]
0x0000002cacf77800 JavaThread "ROW-MUTATION-STAGE:27" [_thread_= blocked, id=3D10805]
0x0000002cacf76000 JavaThread "ROW-MUTATION-= STAGE:26" [_thread_blocked, id=3D10804]
0x0000002cacf74c00 JavaTh= read "ROW-MUTATION-STAGE:25" [_thread_blocked, id=3D10803]
0x0000002cacf73400 JavaThread "ROW-MUTATION-STAGE:24" [_thread_= blocked, id=3D10802]
0x0000002cacf72000 JavaThread "ROW-MUTATION-= STAGE:23" [_thread_blocked, id=3D10801]
0x0000002cacf70c00 JavaTh= read "ROW-MUTATION-STAGE:22" [_thread_blocked, id=3D10800]
0x0000002cacf6f400 JavaThread "ROW-MUTATION-STAGE:21" [_thread_= blocked, id=3D10799]
0x0000002cacf6e000 JavaThread "ROW-MUTATION-= STAGE:20" [_thread_blocked, id=3D10798]
0x0000002cacf6c800 JavaTh= read "ROW-MUTATION-STAGE:19" [_thread_blocked, id=3D10797]
0x0000002cacf6b400 JavaThread "ROW-MUTATION-STAGE:18" [_thread_= blocked, id=3D10796]
0x0000002cacf6a000 JavaThread "ROW-MUTATION-= STAGE:17" [_thread_blocked, id=3D10795]
0x0000002cacf68800 JavaTh= read "ROW-MUTATION-STAGE:16" [_thread_blocked, id=3D10794]
0x0000002cacf67400 JavaThread "ROW-MUTATION-STAGE:15" [_thread_= blocked, id=3D10793]
0x0000002cacf65c00 JavaThread "ROW-MUTATION-= STAGE:14" [_thread_blocked, id=3D10792]
0x0000002cacf64800 JavaTh= read "ROW-MUTATION-STAGE:13" [_thread_blocked, id=3D10791]
0x0000002cacf63400 JavaThread "ROW-MUTATION-STAGE:12" [_thread_= blocked, id=3D10790]
0x0000002cacf61c00 JavaThread "ROW-MUTATION-= STAGE:11" [_thread_blocked, id=3D10789]
0x0000002cacf60800 JavaTh= read "ROW-MUTATION-STAGE:10" [_thread_blocked, id=3D10788]
0x0000002cacf5f000 JavaThread "ROW-MUTATION-STAGE:9" [_thread_b= locked, id=3D10787]
0x0000002cacf5bc00 JavaThread "ROW-MUTATION-S= TAGE:8" [_thread_blocked, id=3D10786]
0x0000002cacf5a400 JavaThre= ad "ROW-MUTATION-STAGE:7" [_thread_blocked, id=3D10785]
0x0000002cacf5d400 JavaThread "ROW-MUTATION-STAGE:6" [_thread_b= locked, id=3D10784]
0x0000002cacf5c400 JavaThread "ROW-MUTATION-S= TAGE:5" [_thread_blocked, id=3D10783]
0x0000002cacf57800 JavaThre= ad "ROW-MUTATION-STAGE:4" [_thread_blocked, id=3D10782]
0x0000002cad258400 JavaThread "ROW-MUTATION-STAGE:3" [_thread_b= locked, id=3D10781]
0x0000002ca98f3400 JavaThread "ROW-MUTATION-S= TAGE:2" [_thread_blocked, id=3D10780]
0x0000002ca98f2400 JavaThread= "ROW-MUTATION-STAGE:1" [_thread_blocked, id=3D10779]
0x0000002ca9902000 JavaThread "Thread-9" [_thread_in_native, id= =3D10778]
0x0000002ca9901800 JavaThread "Thread-8" [_thread_= in_native, id=3D10777]
0x0000002cad253000 JavaThread "Timer-1&quo= t; [_thread_blocked, id=3D10776]
0x0000002cad25ec00 JavaThread "LB-TARGET:1" [_thread_blocked, i= d=3D10775]
0x0000002cad25d800 JavaThread "LB-OPERATIONS:1" [= _thread_blocked, id=3D10774]
0x0000002cad25cc00 JavaThread "ACCEPT-krs1165.nhncorp= .com/10.25.32.35" [_thread_in_native, id=3D10773]
0x0000002cad25bc00 JavaThread "Timer-0" [_thread_blocked, id=3D= 10772]
0x0000002cacf19400 JavaThread "GC inspection" [_threa= d_blocked, id=3D10771]
0x0000002cad28bc00 JavaThread "PERIODIC-CO= MMIT-LOG-SYNCER" [_thread_blocked, id=3D10770]
0x0000002cad2a4400 JavaThread "COMMIT-LOG-WRITER" [_thread_bloc= ked, id=3D10769]
=3D>0x0000002ca9903400 JavaThread "COMPACTION-P= OOL:1" [_thread_in_vm, id=3D10768]
0x0000002cad2a3400 JavaThread = "SSTABLE-CLEANUP-TIMER" [_thread_blocked, id=3D10378]
0x0000002cace74c00 JavaThread "SSTABLE-DELETER" [_thread_blocke= d, id=3D10377]
0x0000002cad227000 JavaThread "MESSAGE-STREAMING-P= OOL:1" [_thread_blocked, id=3D10376]
0x0000002cad225c00 JavaThrea= d "MESSAGE-DESERIALIZER-POOL:1" [_thread_blocked, id=3D10375]
0x0000002cad224800 JavaThread "CACHETABLE-TIMER-2" daemon [_thr= ead_blocked, id=3D10374]
0x0000002cad224000 JavaThread "CACHETABL= E-TIMER-1" daemon [_thread_blocked, id=3D10373]
0x0000002cacf55c0= 0 JavaThread "CONSISTENCY-MANAGER:4" [_thread_blocked, id=3D10372= ]
0x0000002cacfa1800 JavaThread "CONSISTENCY-MANAGER:3" [_thread_= blocked, id=3D10371]
0x0000002cacf9cc00 JavaThread "CONSISTENCY-M= ANAGER:2" [_thread_blocked, id=3D10370]
0x0000002cacfa2800 JavaTh= read "CONSISTENCY-MANAGER:1" [_thread_blocked, id=3D10369]
0x0000002cacf9b000 JavaThread "MEMTABLE-POST-FLUSHER:1" [_threa= d_blocked, id=3D10368]
0x0000002cad230800 JavaThread "FLUSH-WRITE= R-POOL:1" [_thread_blocked, id=3D10367]
0x0000002cad237c00 JavaTh= read "FLUSH-SORTER-POOL:1" [_thread_blocked, id=3D10366]
0x0000002cad230400 JavaThread "FLUSH-TIMER" [_thread_blocked, i= d=3D10365]
0x0000002cacda6800 JavaThread "RMI TCP Accept-8080&quo= t; daemon [_thread_in_native, id=3D10363]
0x0000002cacd48800 JavaThrea= d "RMI TCP Accept-0" daemon [_thread_in_native, id=3D10362]
0x0000002ca9893400 JavaThread "Low Memory Detector" daemon [_th= read_blocked, id=3D10361]
0x0000002ca9891400 JavaThread "Compiler= Thread1" daemon [_thread_blocked, id=3D10360]
0x0000002ca988f800 = JavaThread "CompilerThread0" daemon [_thread_blocked, id=3D10359]=
0x0000002ca988e000 JavaThread "Signal Dispatcher" daemon [_thre= ad_blocked, id=3D10358]
0x0000002ca988c800 JavaThread "Surrogate = Locker Thread (CMS)" daemon [_thread_blocked, id=3D10357]
0x00000= 02ca9869c00 JavaThread "Finalizer" daemon [_thread_blocked, id=3D= 10356]
0x0000002ca9869000 JavaThread "Reference Handler" daemon [_thre= ad_blocked, id=3D10355]
0x0000000040112800 JavaThread "main"= [_thread_in_native, id=3D10348]

Other Threads:
0x0000002ca9864= 400 VMThread [id=3D10354]
0x0000002cacd8d800 WatcherThread [id=3D10364]

VM state:not at safe= point (normal execution)

VM Mutex/Monitor currently owned by a threa= d: None

Heap
par new generation total 76672K, used 55139K [0x0= 000002a995d0000, 0x0000002a9e900000, 0x0000002a9e900000)
eden space 68160K, 68% used [0x0000002a995d0000, 0x0000002a9c359758, 0x0= 000002a9d860000)
from space 8512K, 99% used [0x0000002a9e0b0000, 0x00= 00002a9e8ff820, 0x0000002a9e900000)
to space 8512K, 0% used [0x000= 0002a9d860000, 0x0000002a9d860000, 0x0000002a9e0b0000)
concurrent mark-sweep generation total 5810916K, used 4774431K [0x0000002a= 9e900000, 0x0000002c013b9000, 0x0000002c995d0000)
concurrent-mark-sweep= perm gen total 26096K, used 15598K [0x0000002c995d0000, 0x0000002c9af4c000= , 0x0000002c9e9d0000)



....(the rest omitt= ed)




--------= ------- S Y S T E M ---------------

OS:CentOS release 4.5 (Final)<= br>
uname:Linux 2.6.9-55.0.12.EL.nhnsmp #1 SMP Tue Nov 6 18:38:43 KST 2007 = x86_64
libc:glibc 2.3.4 NPTL 2.3.4
rlimit: STACK 10240k, CORE 0k, NPR= OC 16383, NOFILE 32768, AS infinity
load average:1.74 1.51 1.44

CPU:total 4 (2 cores per cpu, 1 threads per core) family 6 model 15 steppin= g 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, = physical 16407020k(51816k free), swap 4192956k(4016836k free)

vm_inf= o: Java HotSpot(TM) 64-Bit Server VM (1.6.0_03-b05) for linux-amd64, built = on Sep 24 2007 23:09:21 by "java_re" with gcc 3.2.2 (SuSE Linux)<= br>



I set java opti= on -Xmx8G and the others were unchanged from Cassandra 0.6.3 settings.


I will try upgrading JVM first.

Thanks for advice. :)







2010/8/3 Jonathan Ellis <= jbellis@gmail.com>
If you have a crash log you should post at = least the header rather
than playing 20 questions with us. =A0But if it's not OOM then it's=
likely to be a bug in the JVM, so upgrading is probably your best
option.

On Tue, Aug 3, 2010 at 3:49 AM, Ilun Ahn <prozect.mail@gmail.com> wrote:
> No, I don't think the direct cause is out of heap space. It didn&#= 39;t left any
> heap dump file with the option -XX:+HeapDumpOnOutOfMemoryError.
> My system.log for the last minute is as follows(many GC occurs):
> INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,254
> HintedHandOffManager.java (line 153) Started hinted handoff for endPoi= nt
> /10.25.32.36
> INFO [HINTED-HANDOFF-POOL:1] 2010-08-02 20:33:50,255
> HintedHandOffManager.java (line 210) Finished hinted handoff of 0 rows= to
> endpoint /10.25.32.36=
> INFO [GC inspection] 2010-08-02 20:34:01,919 GCInspector.java (line 11= 0) GC
> for ParNew: 269 ms, 11161808 reclaimed leaving 5068544312 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:34:14,985 GCInspector.java (line 11= 0) GC
> for ParNew: 208 ms, 12326736 reclaimed leaving 4044195008 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:34:15,986 GCInspector.java (line 11= 0) GC
> for ParNew: 208 ms, 12283112 reclaimed leaving 2005777224 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:35:45,834 GCInspector.java (line 11= 0) GC
> for ParNew: 229 ms, 13074080 reclaimed leaving 5374833480 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:35:46,836 GCInspector.java (line 11= 0) GC
> for ParNew: 203 ms, 12529824 reclaimed leaving 5321733432 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:37:01,624 GCInspector.java (line 11= 0) GC
> for ParNew: 206 ms, 11029656 reclaimed leaving 4473650352 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:38:19,064 GCInspector.java (line 11= 0) GC
> for ConcurrentMarkSweep: 4501 ms, 1057548400 reclaimed leaving 2461458= 096
> used; max is 8719630336
> INFO [GC inspection] 2010-08-02 20:41:20,446 GCInspector.java (line 11= 0) GC
> for ParNew: 218 ms, 15072720 reclaimed leaving 5345683640 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:41:23,453 GCInspector.java (line 11= 0) GC
> for ParNew: 234 ms, 16818048 reclaimed leaving 3937902088 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:42:15,229 GCInspector.java (line 11= 0) GC
> for ConcurrentMarkSweep: 8015 ms, 739534984 reclaimed leaving 35501380= 24
> used; max is 8719630336
> INFO [GC inspection] 2010-08-02 20:42:42,444 GCInspector.java (line 11= 0) GC
> for ParNew: 203 ms, 14218928 reclaimed leaving 4398967608 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:42:43,565 GCInspector.java (line 11= 0) GC
> for ParNew: 203 ms, 12274600 reclaimed leaving 1989854648 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:43:01,801 GCInspector.java (line 11= 0) GC
> for ParNew: 212 ms, 10183184 reclaimed leaving 2337034168 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:44:01,215 GCInspector.java (line 11= 0) GC
> for ParNew: 218 ms, 10402368 reclaimed leaving 4334140184 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:44:35,623 GCInspector.java (line 11= 0) GC
> for ConcurrentMarkSweep: 4424 ms, 3101007888 reclaimed leaving 2459621= 048
> used; max is 8719630336
> INFO [GC inspection] 2010-08-02 20:45:32,089 GCInspector.java (line 11= 0) GC
> for ParNew: 227 ms, 27109720 reclaimed leaving 5410486832 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:45:35,095 GCInspector.java (line 11= 0) GC
> for ParNew: 203 ms, 28235832 reclaimed leaving 3580093424 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:46:01,203 GCInspector.java (line 11= 0) GC
> for ParNew: 257 ms, 12257744 reclaimed leaving 3469012312 used; max is=
> 8719630336
> INFO [GC inspection] 2010-08-02 20:46:51,060 GCInspector.java (line 11= 0) GC
> for ParNew: 222 ms, 18473064 reclaimed leaving 5320004640 used; max is=
> 8719630336
>
>
>
>
>
> and the JRE crash log show that current thread is COMPACTION-POOL:1. >
>
>
> ps. Sorry for unnecessary message. my mistake.
>
>
>
>
> 2010/8/3 Ilun Ahn <prozec= t.mail@gmail.com>
>>
>>
>> 2010/8/2 Peter Schuller <peter.schuller@infidyne.com>
>>>
>>> > First, Cassandra suddenly dies during compaction. Java co= re dump says
>>> > that
>>> > the last thread run was=A0 "COMPACTION-POOL:1".=
>>> > I suspect that my business logic could lead size of colum= ns in a column
>>> > family per a row to be greater than two gigabytes. (but i= couldn't
>>> > confirm
>>> > it yet)
>>>
>>> Are you running out of memory (java heap)? If you're runni= ng cassandra
>>> with default options, it will be running with
>>> -XX:+HeapDumpOnOutOfMemoryError
>>>
>>> Have you checked the cassandra system.log for garbage collecti= on
>>> messages? What is in the last minute or two of logs?
>>>
>>> --
>>> / Peter Schuller
>>
>
>



--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

--001485f5b188a72830048cf5bd0e--