Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 59A431048F for ; Tue, 6 Jan 2015 07:26:33 +0000 (UTC) Received: (qmail 51576 invoked by uid 500); 6 Jan 2015 07:26:21 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 51535 invoked by uid 500); 6 Jan 2015 07:26:21 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 51524 invoked by uid 99); 6 Jan 2015 07:26:21 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jan 2015 07:26:21 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,SPF_PASS,T_REMOTE_IMAGE X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of lijianqi@jd.com designates 58.83.206.59 as permitted sender) Received: from [58.83.206.59] (HELO smtp.jd.com) (58.83.206.59) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Jan 2015 07:26:16 +0000 Received: from YYXXLIJIANQIPC (103.16.127.6) by HUB03.360buyAD.local (172.17.27.17) with Microsoft SMTP Server id 14.3.123.3; Tue, 6 Jan 2015 15:25:44 +0800 From: =?UTF-8?B?5p2O5bu65aWH?= To: References: <424E80B0-A06C-4CBB-AA66-93A020A074DC@mac.com> <50F96AC3-D9E2-4640-BE79-AEDC3069BF61@mac.com> In-Reply-To: Subject: =?UTF-8?Q?=E7=AD=94=E5=A4=8D:_Cassandra_consuming_whole_RA?= =?UTF-8?Q?M_=2864_G=29?= Date: Tue, 6 Jan 2015 15:25:44 +0800 Message-ID: <049801d02981$fbb3a9c0$f31afd40$@jd.com> MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_0499_01D029C5.09E122F0" X-Mailer: Microsoft Outlook 14.0 Thread-Index: AQKAHy8MJ73xpQzre1AdOJLoX/h+9AFb9OlzAoAbyDIBa5lEEwIXPQWbAO7HA/QCxj9QiwLw3fXMmuKdtLA= Content-Language: zh-cn X-Originating-IP: [103.16.127.6] X-Virus-Checked: Checked by ClamAV on apache.org ------=_NextPart_000_0499_01D029C5.09E122F0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi , There is a =E2=80=9Cpossible memory leak =E2=80=9C issue with c* = 2.1.2. https://issues.apache.org/jira/browse/CASSANDRA-8248 It happen with our c* 2.1.2 cluster . In /proc/{pid}/maps there are a = lot of deleted file maps . =20 =20 _____ =20 Stephen li=20 =20 =20 =E5=8F=91=E4=BB=B6=E4=BA=BA: Joe Ramsey [mailto:joe.ramsey@mac.com]=20 =E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2015=E5=B9=B41=E6=9C=886=E6=97=A5 = 15:12 =E6=94=B6=E4=BB=B6=E4=BA=BA: user@cassandra.apache.org =E4=B8=BB=E9=A2=98: Re: Cassandra consuming whole RAM (64 G) =20 Thanks Rahul and good luck! I=E2=80=99m really curious to hear what the = result is. =20 =20 On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj = wrote: =20 Thanks for your response.. i will get back to you with my findings. =20 On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey wrote: That should be =E2=80=9Cwriting too many bytes=E2=80=9D not = =E2=80=9Cwaiting too many bytes=E2=80=9D just for clarity=E2=80=99s = sake. =20 On Jan 6, 2015, at 2:03 AM, Joe Ramsey wrote: =20 I=E2=80=99m not an expert. Really just learning this myself but it = looks like according to the stack you=E2=80=99re getting an exception = waiting too many bytes to the commit log. =20 =20 That=E2=80=99s controlled by commit_log_segment_size_in_mb setting. The = maximum write size that C* will allow is half of the value set for this = parameter so if it=E2=80=99s set for 32 (default) that means your max = write would be 16MB. (And that=E2=80=99s what=E2=80=99s getting = reported in the stack trace. You=E2=80=99re writing 16965030 bytes = (16.18M) and your max write is 16MB so it=E2=80=99s throwing the = exception. Try setting commit_log_segment_size_in_mb from 32 to 64 and = see if the error goes away. It really should get tuned for the amount = of writes but that=E2=80=99ll tell you at least if this is the issue. =20 Let me know how it goes! =20 Joe =20 =20 =20 On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj = wrote: =20 Hi Joe, =20 PFB output of system.log =20 tail -n 100 system.log INFO [CompactionExecutor:164] 2015-01-06 11:58:28,555 = CompactionTask.java:251 - Compacted 4 sstables to = [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f= 1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-2= 43,]. 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms = =3D 0.073765MB/s. 277,719 total partitions merged to 260,285. = Partition merge counts were {1:246186, 2:11302, 3:2259, 4:538, } INFO [CompactionExecutor:176] 2015-01-06 11:58:28,579 = CompactionTask.java:251 - Compacted 4 sstables to = [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c38869= 0a4acb25fe1f77b/system-compactions_in_progress-ka-882,]. 511 bytes to = 42 (~8% of original) in 27ms =3D 0.001483MB/s. 7 total partitions = merged to 1. Partition merge counts were {1:1, 2:3, } WARN [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - = Batch of prepared statements for [clickstream.business_feed] is of size = 433053, exceeding specified threshold of 5120 by 427933. INFO [CompactionExecutor:175] 2015-01-06 11:59:19,883 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')] INFO [CompactionExecutor:175] 2015-01-06 11:59:19,952 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5647,]. 2,801,126 bytes to 2,801,126 (~100% of = original) in 67ms =3D 39.871073MB/s. 26 total partitions merged to 26. = Partition merge counts were {1:26, } WARN [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch = of prepared statements for [clickstream.business_feed] is of size = 433819, exceeding specified threshold of 5120 by 428699. INFO [CompactionExecutor:178] 2015-01-06 12:00:19,982 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')] INFO [CompactionExecutor:178] 2015-01-06 12:00:20,048 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5648,]. 2,801,126 bytes to 2,801,126 (~100% of = original) in 65ms =3D 41.097876MB/s. 26 total partitions merged to 26. = Partition merge counts were {1:26, } WARN [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch = of prepared statements for [clickstream.business_feed] is of size = 436755, exceeding specified threshold of 5120 by 431635. INFO [CompactionExecutor:179] 2015-01-06 12:01:20,074 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')] INFO [CompactionExecutor:179] 2015-01-06 12:01:20,139 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5649,]. 2,801,126 bytes to 2,801,126 (~100% of = original) in 63ms =3D 42.402570MB/s. 26 total partitions merged to 26. = Partition merge counts were {1:26, } WARN [SharedPool-Worker-1] 2015-01-06 12:01:29,692 = AbstractTracingAwareExecutorService.java:169 - Uncaught exception on = thread Thread[SharedPool-Worker-1,5,main]: {} java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too = large for the maxiumum size of 16777216 at = org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.ja= va:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown = Source) ~[na:1.7.0_71] at = org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Futur= eTask.run(AbstractTracingAwareExecutorService.java:164) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) = [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] INFO [CompactionExecutor:180] 2015-01-06 12:02:20,167 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')] INFO [CompactionExecutor:180] 2015-01-06 12:02:20,241 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5650,]. 2,801,126 bytes to 2,801,126 (~100% of = original) in 72ms =3D 37.102249MB/s. 26 total partitions merged to 26. = Partition merge counts were {1:26, } INFO [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 - = ParNew GC in 380ms. CMS Old Gen: 1551865712 -> 1704414952; Par Eden = Space: 1718091776 -> 0; Par Survivor Space: 161099424 -> 214695936 WARN [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch = of prepared statements for = [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, = exceeding specified threshold of 5120 by 54901. INFO [CompactionExecutor:181] 2015-01-06 12:03:20,274 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')] INFO [CompactionExecutor:181] 2015-01-06 12:03:20,348 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5651,]. 2,801,126 bytes to 2,801,126 (~100% of = original) in 73ms =3D 36.593999MB/s. 26 total partitions merged to 26. = Partition merge counts were {1:26, } WARN [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch = of prepared statements for [clickstream.business_feed] is of size = 517769, exceeding specified threshold of 5120 by 512649. INFO [CompactionExecutor:182] 2015-01-06 12:04:20,374 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')] INFO [CompactionExecutor:182] 2015-01-06 12:04:20,445 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5652,]. 2,801,126 bytes to 2,801,126 (~100% of = original) in 69ms =3D 38.715390MB/s. 26 total partitions merged to 26. = Partition merge counts were {1:26, } WARN [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - = Batch of prepared statements for [clickstream.im_csl_log] is of size = 13220136, exceeding specified threshold of 5120 by 13215016. WARN [SharedPool-Worker-1] 2015-01-06 12:04:25,092 = AbstractTracingAwareExecutorService.java:169 - Uncaught exception on = thread Thread[SharedPool-Worker-1,5,main]: {} java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation = of 16800887 bytes is too large for the maxiumum size of 16777216 at = org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(Stora= geProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown = Source) ~[na:1.7.0_71] at = org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Futur= eTask.run(AbstractTracingAwareExecutorService.java:164) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) = [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 = bytes is too large for the maxiumum size of 16777216 at = org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java= :999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(Stora= geProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] ... 4 common frames omitted INFO [CompactionExecutor:183] 2015-01-06 12:05:20,472 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')] INFO [CompactionExecutor:183] 2015-01-06 12:05:20,538 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5653,]. 2,801,126 bytes to 2,801,126 (~100% of = original) in 64ms =3D 41.740030MB/s. 26 total partitions merged to 26. = Partition merge counts were {1:26, } WARN [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch = of prepared statements for [clickstream.im_csl_log] is of size 13136950, = exceeding specified threshold of 5120 by 13131830. INFO [SlabPoolCleaner] 2015-01-06 12:05:26,129 = ColumnFamilyStore.java:840 - Enqueuing flush of im_csl_log: 505903980 = (24%) on-heap, 0 (0%) off-heap INFO [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325 = - Writing Memtable-im_csl_log@1246880835(90297505 serialized bytes, = 1530359 ops, 20%/0% of on/off-heap limit) INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364 = - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log-ka-2654-Data.db (21516225 bytes) for = commitlog position ReplayPosition(segmentId=3D1420516895898, = position=3D16708232) INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325 = - Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 = serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364 = - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db = (361277 bytes) for commitlog position = ReplayPosition(segmentId=3D1420516895898, position=3D16708232) INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325 = - Writing = Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(211= 0840 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364 = - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-= 2459-Data.db (543861 bytes) for commitlog position = ReplayPosition(segmentId=3D1420516895898, position=3D16708232) INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325 = - Writing = Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(21108= 40 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364 = - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-24= 5-Data.db (481902 bytes) for commitlog position = ReplayPosition(segmentId=3D1420516895898, position=3D16708232) INFO [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325 = - Writing = Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840 = serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364 = - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Dat= a.db (460053 bytes) for commitlog position = ReplayPosition(segmentId=3D1420516895898, position=3D16708232) INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325 = - Writing = Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(211084= 0 serialized bytes, 52771 ops, 1%/0% of on/off-heap limit) INFO [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364 = - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-283= 1-Data.db (395084 bytes) for commitlog position = ReplayPosition(segmentId=3D1420516895898, position=3D16708232) INFO [CompactionExecutor:185] 2015-01-06 12:06:20,563 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')] INFO [CompactionExecutor:185] 2015-01-06 12:06:20,633 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5654,]. 2,801,126 bytes to 2,801,126 (~100% of = original) in 68ms =3D 39.284734MB/s. 26 total partitions merged to 26. = Partition merge counts were {1:26, } INFO [CompactionExecutor:187] 2015-01-06 12:07:20,659 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')] INFO [CompactionExecutor:187] 2015-01-06 12:07:20,733 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5655,]. 2,801,126 bytes to 2,801,126 (~100% of = original) in 72ms =3D 37.102249MB/s. 26 total partitions merged to 26. = Partition merge counts were {1:26, } WARN [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch = of prepared statements for [clickstream.im_csl_log] is of size 13480721, = exceeding specified threshold of 5120 by 13475601. WARN [SharedPool-Worker-1] 2015-01-06 12:07:26,537 = AbstractTracingAwareExecutorService.java:169 - Uncaught exception on = thread Thread[SharedPool-Worker-1,5,main]: {} java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation = of 17138032 bytes is too large for the maxiumum size of 16777216 at = org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(Stora= geProxy.java:2121) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown = Source) ~[na:1.7.0_71] at = org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Futur= eTask.run(AbstractTracingAwareExecutorService.java:164) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) = [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 = bytes is too large for the maxiumum size of 16777216 at = org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java= :999) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(Stora= geProxy.java:2117) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] ... 4 common frames omitted INFO [CompactionExecutor:188] 2015-01-06 12:08:20,762 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')] INFO [CompactionExecutor:188] 2015-01-06 12:08:20,825 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5656,]. 2,801,126 bytes to 2,801,126 (~100% of = original) in 60ms =3D 44.522699MB/s. 26 total partitions merged to 26. = Partition merge counts were {1:26, } INFO [CompactionExecutor:189] 2015-01-06 12:09:20,853 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')] INFO [CompactionExecutor:189] 2015-01-06 12:09:20,921 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5657,]. 2,801,126 bytes to 2,801,126 (~100% of = original) in 66ms =3D 40.475181MB/s. 26 total partitions merged to 26. = Partition merge counts were {1:26, } WARN [SharedPool-Worker-1] 2015-01-06 12:09:22,532 = AbstractTracingAwareExecutorService.java:169 - Uncaught exception on = thread Thread[SharedPool-Worker-1,5,main]: {} java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too = large for the maxiumum size of 16777216 at = org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.ja= va:54) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown = Source) ~[na:1.7.0_71] at = org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Futur= eTask.run(AbstractTracingAwareExecutorService.java:164) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at = org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) = [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT] at java.lang.Thread.run(Unknown Source) [na:1.7.0_71] =20 =20 =20 Regards: Rahul Bhardwaj =20 On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj = wrote: Hi Joe, =20 PFA heap dump =20 =20 regards: Rahul Bhardwaj =20 =20 =20 On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey wrote: Did you try generating a heap dump so you can look through it to see = what=E2=80=99s actually happened? =20 =20 On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj = wrote: =20 Hi, =20 We are using cassandra 2.1 version in a cluster of three machines each = with 64 GB RAM =20 The processes are killed by kernel, coz they are eating all memory = (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) = because we have 64 GB RAM. =20 Please help. =20 =20 Regards: Rahul Bhardwaj Follow IndiaMART.com for latest updates on = this and more: = = Mobile Channel: = = = =20 Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki = Kaam Yahin Banta Hai !!! =20 =20 =20 Follow IndiaMART.com for latest updates on = this and more: = = Mobile Channel: = = = =20 Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki = Kaam Yahin Banta Hai !!! =20 =20 =20 Follow IndiaMART.com for latest updates on = this and more: = = Mobile Channel: = = = =20 Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki = Kaam Yahin Banta Hai !!! =20 ------=_NextPart_000_0499_01D029C5.09E122F0 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hi ,

=C2=A0=C2=A0=C2=A0=C2=A0 There is a =E2=80=9Cpossible memory leak = =E2=80=9C issue with c* 2.1.2.=C2=A0=C2=A0 https://iss= ues.apache.org/jira/browse/CASSANDRA-8248

It happen with our c* 2.1.2 cluster . =C2=A0=C2=A0In /proc/{pid}/maps there are a lot of deleted file maps = .

=C2=A0=C2=A0

 

Stephen li

 

 

=E5=8F=91=E4=BB=B6=E4=BA=BA: Joe Ramsey [mailto:joe.ramsey@mac.com] =
=E5=8F=91=E9=80=81=E6=97=B6=E9=97=B4: 2015=E5=B9=B41=E6=9C=886=E6=97=A5 15:12
=E6=94=B6=E4=BB=B6=E4=BA=BA: = user@cassandra.apache.org
=E4=B8=BB=E9=A2=98: Re: Cassandra consuming = whole RAM (64 G)

 

Thanks Rahul and good luck! =  I=E2=80=99m really curious to hear what the result = is.

 

 

On Jan 6, 2015, at 2:10 AM, Rahul = Bhardwaj <rahul.bhardwaj@indiamart.com= > wrote:

 

Thanks for your response.. i will = get back to you with my findings.

 

On Tue, Jan 6, 2015 at 12:36 PM, = Joe Ramsey <joe.ramsey@mac.com> = wrote:

That should be =E2=80=9Cwriting too many bytes=E2=80=9D not = =E2=80=9Cwaiting too many bytes=E2=80=9D just for clarity=E2=80=99s = sake.

 

On Jan 6, 2015, at 2:03 AM, Joe = Ramsey <joe.ramsey@mac.com> = wrote:

 

I=E2=80=99m not an expert.  = Really just learning this myself but it looks like according to the = stack you=E2=80=99re getting an exception waiting too many bytes to the = commit log.  

 

That=E2=80=99s controlled by = commit_log_segment_size_in_mb setting.  The maximum write size that = C* will allow is half of the value set for this parameter so if = it=E2=80=99s set for 32 (default) that means your max write would be = 16MB.  (And that=E2=80=99s what=E2=80=99s getting reported in the = stack trace.  You=E2=80=99re writing 16965030 bytes (16.18M) and = your max write is 16MB so it=E2=80=99s throwing the exception.  Try = setting commit_log_segment_size_in_mb from 32 to 64 and see if the error = goes away.  It really should get tuned for the amount of writes but = that=E2=80=99ll tell you at least if this is the = issue.

 

Let me know how it = goes!

 

Joe

 

 

 

On Jan 6, 2015, at 1:40 AM, Rahul = Bhardwaj <rahul.bhardwaj@indiamart.com> = wrote:

 

Hi = Joe,

 

PFB output of = system.log

 

 tail -n 100 = system.log

INFO  [CompactionExecutor:164] 2015-01-06 11:58:28,555 = CompactionTask.java:251 - Compacted 4 sstables to = [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f= 1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-2= 43,].  50,514,079 bytes to 50,626,490 (~100% of original) in = 654,529ms =3D 0.073765MB/s.  277,719 total partitions merged to = 260,285.  Partition merge counts were {1:246186, 2:11302, 3:2259, = 4:538, }

INFO  [CompactionExecutor:176] 2015-01-06 11:58:28,579 = CompactionTask.java:251 - Compacted 4 sstables to = [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c38869= 0a4acb25fe1f77b/system-compactions_in_progress-ka-882,].  511 bytes = to 42 (~8% of original) in 27ms =3D 0.001483MB/s.  7 total = partitions merged to 1.  Partition merge counts were {1:1, 2:3, = }

WARN  [Thrift:16] 2015-01-06 11:59:02,351 = BatchStatement.java:255 - Batch of prepared statements for = [clickstream.business_feed] is of size 433053, exceeding specified = threshold of 5120 by 427933.

INFO  [CompactionExecutor:175] = 2015-01-06 11:59:19,883 CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]

INFO =  [CompactionExecutor:175] 2015-01-06 11:59:19,952 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5647,].  2,801,126 bytes to 2,801,126 (~100% of = original) in 67ms =3D 39.871073MB/s.  26 total partitions merged to = 26.  Partition merge counts were {1:26, = }

WARN  [Thrift:4] 2015-01-06 12:00:02,457 = BatchStatement.java:255 - Batch of prepared statements for = [clickstream.business_feed] is of size 433819, exceeding specified = threshold of 5120 by 428699.

INFO  [CompactionExecutor:178] = 2015-01-06 12:00:19,982 CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]

INFO =  [CompactionExecutor:178] 2015-01-06 12:00:20,048 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5648,].  2,801,126 bytes to 2,801,126 (~100% of = original) in 65ms =3D 41.097876MB/s.  26 total partitions merged to = 26.  Partition merge counts were {1:26, = }

WARN  [Thrift:5] 2015-01-06 12:01:02,387 = BatchStatement.java:255 - Batch of prepared statements for = [clickstream.business_feed] is of size 436755, exceeding specified = threshold of 5120 by 431635.

INFO  [CompactionExecutor:179] = 2015-01-06 12:01:20,074 CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]

INFO =  [CompactionExecutor:179] 2015-01-06 12:01:20,139 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5649,].  2,801,126 bytes to 2,801,126 (~100% of = original) in 63ms =3D 42.402570MB/s.  26 total partitions merged to = 26.  Partition merge counts were {1:26, = }

WARN  [SharedPool-Worker-1] 2015-01-06 12:01:29,692 = AbstractTracingAwareExecutorService.java:169 - Uncaught exception on = thread Thread[SharedPool-Worker-1,5,main]: = {}

java.lang.IllegalArgumentException: Mutation of 17225046 = bytes is too large for the maxiumum size of = 16777216

        at = org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at = org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.ja= va:54) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at = org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :62) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) = ~[na:1.7.0_71]

        at = org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Futur= eTask.run(AbstractTracingAwareExecutorService.java:164) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) = [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        = at java.lang.Thread.run(Unknown Source) = [na:1.7.0_71]

INFO  [CompactionExecutor:180] 2015-01-06 12:02:20,167 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]

INFO =  [CompactionExecutor:180] 2015-01-06 12:02:20,241 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5650,].  2,801,126 bytes to 2,801,126 (~100% of = original) in 72ms =3D 37.102249MB/s.  26 total partitions merged to = 26.  Partition merge counts were {1:26, = }

INFO  [Service Thread] 2015-01-06 12:03:02,108 = GCInspector.java:142 - ParNew GC in 380ms.  CMS Old Gen: 1551865712 = -> 1704414952; Par Eden Space: 1718091776 -> 0; Par Survivor = Space: 161099424 -> 214695936

WARN  [Thrift:4] 2015-01-06 = 12:03:03,357 BatchStatement.java:255 - Batch of prepared statements for = [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, = exceeding specified threshold of 5120 by = 54901.

INFO  [CompactionExecutor:181] 2015-01-06 12:03:20,274 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]

INFO =  [CompactionExecutor:181] 2015-01-06 12:03:20,348 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5651,].  2,801,126 bytes to 2,801,126 (~100% of = original) in 73ms =3D 36.593999MB/s.  26 total partitions merged to = 26.  Partition merge counts were {1:26, = }

WARN  [Thrift:3] 2015-01-06 12:04:02,524 = BatchStatement.java:255 - Batch of prepared statements for = [clickstream.business_feed] is of size 517769, exceeding specified = threshold of 5120 by 512649.

INFO  [CompactionExecutor:182] = 2015-01-06 12:04:20,374 CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]

INFO =  [CompactionExecutor:182] 2015-01-06 12:04:20,445 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5652,].  2,801,126 bytes to 2,801,126 (~100% of = original) in 69ms =3D 38.715390MB/s.  26 total partitions merged to = 26.  Partition merge counts were {1:26, = }

WARN  [Thrift:10] 2015-01-06 12:04:24,967 = BatchStatement.java:255 - Batch of prepared statements for = [clickstream.im_csl_log] is of size 13220136, exceeding specified = threshold of 5120 by 13215016.

WARN  [SharedPool-Worker-1] = 2015-01-06 12:04:25,092 AbstractTracingAwareExecutorService.java:169 - = Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: = {}

java.lang.RuntimeException: = java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too = large for the maxiumum size of = 16777216

        at = org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(Stora= geProxy.java:2121) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) = ~[na:1.7.0_71]

        at = org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Futur= eTask.run(AbstractTracingAwareExecutorService.java:164) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) = [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        = at java.lang.Thread.run(Unknown Source) = [na:1.7.0_71]

Caused by: java.lang.IllegalArgumentException: Mutation of = 16800887 bytes is too large for the maxiumum size of = 16777216

        at = org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at = org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java= :999) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at = org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(Stora= geProxy.java:2117) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = ... 4 common frames omitted

INFO  [CompactionExecutor:183] = 2015-01-06 12:05:20,472 CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]

INFO =  [CompactionExecutor:183] 2015-01-06 12:05:20,538 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5653,].  2,801,126 bytes to 2,801,126 (~100% of = original) in 64ms =3D 41.740030MB/s.  26 total partitions merged to = 26.  Partition merge counts were {1:26, = }

WARN  [Thrift:8] 2015-01-06 12:05:24,982 = BatchStatement.java:255 - Batch of prepared statements for = [clickstream.im_csl_log] is of size 13136950, exceeding specified = threshold of 5120 by 13131830.

INFO  [SlabPoolCleaner] = 2015-01-06 12:05:26,129 ColumnFamilyStore.java:840 - Enqueuing flush of = im_csl_log: 505903980 (24%) on-heap, 0 (0%) = off-heap

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 = Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505 = serialized bytes, 1530359 ops, 20%/0% of on/off-heap = limit)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 = Memtable.java:364 - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log-ka-2654-Data.db (21516225 bytes) for = commitlog position ReplayPosition(segmentId=3D1420516895898, = position=3D16708232)

INFO  [MemtableFlushWriter:44] = 2015-01-06 12:05:28,625 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840 = serialized bytes, 52771 ops, 1%/0% of on/off-heap = limit)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 = Memtable.java:364 - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db = (361277 bytes) for commitlog position = ReplayPosition(segmentId=3D1420516895898, = position=3D16708232)

INFO  [MemtableFlushWriter:44] = 2015-01-06 12:05:28,724 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_em= ail_idx@823892581(2110840 serialized bytes, 52771 ops, 1%/0% of = on/off-heap limit)

INFO  [MemtableFlushWriter:44] = 2015-01-06 12:05:28,848 Memtable.java:364 - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-= 2459-Data.db (543861 bytes) for commitlog position = ReplayPosition(segmentId=3D1420516895898, = position=3D16708232)

INFO  [MemtableFlushWriter:44] = 2015-01-06 12:05:28,850 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_= idx@188053011(2110840 serialized bytes, 52771 ops, 1%/0% of = on/off-heap limit)

INFO  [MemtableFlushWriter:44] = 2015-01-06 12:05:28,988 Memtable.java:364 - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-24= 5-Data.db (481902 bytes) for commitlog position = ReplayPosition(segmentId=3D1420516895898, = position=3D16708232)

INFO  [MemtableFlushWriter:44] = 2015-01-06 12:05:28,989 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511= 140(2110840 serialized bytes, 52771 ops, 1%/0% of on/off-heap = limit)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 = Memtable.java:364 - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Dat= a.db (460053 bytes) for commitlog position = ReplayPosition(segmentId=3D1420516895898, = position=3D16708232)

INFO  [MemtableFlushWriter:44] = 2015-01-06 12:05:29,103 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_id= x@258991920(2110840 serialized bytes, 52771 ops, 1%/0% of = on/off-heap limit)

INFO  [MemtableFlushWriter:44] = 2015-01-06 12:05:29,211 Memtable.java:364 - Completed flushing = /var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1= e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-283= 1-Data.db (395084 bytes) for commitlog position = ReplayPosition(segmentId=3D1420516895898, = position=3D16708232)

INFO  [CompactionExecutor:185] = 2015-01-06 12:06:20,563 CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]

INFO =  [CompactionExecutor:185] 2015-01-06 12:06:20,633 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5654,].  2,801,126 bytes to 2,801,126 (~100% of = original) in 68ms =3D 39.284734MB/s.  26 total partitions merged to = 26.  Partition merge counts were {1:26, = }

INFO  [CompactionExecutor:187] 2015-01-06 12:07:20,659 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]

INFO =  [CompactionExecutor:187] 2015-01-06 12:07:20,733 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5655,].  2,801,126 bytes to 2,801,126 (~100% of = original) in 72ms =3D 37.102249MB/s.  26 total partitions merged to = 26.  Partition merge counts were {1:26, = }

WARN  [Thrift:6] 2015-01-06 12:07:26,364 = BatchStatement.java:255 - Batch of prepared statements for = [clickstream.im_csl_log] is of size 13480721, exceeding specified = threshold of 5120 by 13475601.

WARN  [SharedPool-Worker-1] = 2015-01-06 12:07:26,537 AbstractTracingAwareExecutorService.java:169 - = Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: = {}

java.lang.RuntimeException: = java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too = large for the maxiumum size of = 16777216

        at = org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(Stora= geProxy.java:2121) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) = ~[na:1.7.0_71]

        at = org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Futur= eTask.run(AbstractTracingAwareExecutorService.java:164) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) = [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        = at java.lang.Thread.run(Unknown Source) = [na:1.7.0_71]

Caused by: java.lang.IllegalArgumentException: Mutation of = 17138032 bytes is too large for the maxiumum size of = 16777216

        at = org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at = org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java= :999) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at = org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(Stora= geProxy.java:2117) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = ... 4 common frames omitted

INFO  [CompactionExecutor:188] = 2015-01-06 12:08:20,762 CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]

INFO =  [CompactionExecutor:188] 2015-01-06 12:08:20,825 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5656,].  2,801,126 bytes to 2,801,126 (~100% of = original) in 60ms =3D 44.522699MB/s.  26 total partitions merged to = 26.  Partition merge counts were {1:26, = }

INFO  [CompactionExecutor:189] 2015-01-06 12:09:20,853 = CompactionTask.java:136 - Compacting = [SSTableReader(path=3D'/var/lib/cassandra/data/system/batchlog-0290003c97= 7e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]

INFO =  [CompactionExecutor:189] 2015-01-06 12:09:20,921 = CompactionTask.java:251 - Compacted 1 sstables to = [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b= /system-batchlog-ka-5657,].  2,801,126 bytes to 2,801,126 (~100% of = original) in 66ms =3D 40.475181MB/s.  26 total partitions merged to = 26.  Partition merge counts were {1:26, = }

WARN  [SharedPool-Worker-1] 2015-01-06 12:09:22,532 = AbstractTracingAwareExecutorService.java:169 - Uncaught exception on = thread Thread[SharedPool-Worker-1,5,main]: = {}

java.lang.IllegalArgumentException: Mutation of 16965030 = bytes is too large for the maxiumum size of = 16777216

        at = org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at = org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.ja= va:54) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at = org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java= :62) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) = ~[na:1.7.0_71]

        at = org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$Futur= eTask.run(AbstractTracingAwareExecutorService.java:164) = ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

<= div>

        = at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) = [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        = at java.lang.Thread.run(Unknown Source) = [na:1.7.0_71]

 

 

 

Regards:

Rahul = Bhardwaj

 

On Tue, Jan 6, 2015 at 12:08 PM, = Rahul Bhardwaj <rahul.bhardwaj@indiamart.com> = wrote:

Hi Joe,

 

PFA heap = dump

 

 

regards:

Rahul = Bhardwaj

 

 

 

On Tue, Jan 6, 2015 at 11:35 AM, = Joe Ramsey <joe.ramsey@mac.com> = wrote:

Did you try generating a heap dump so you can look through = it to see what=E2=80=99s actually happened?

 

 

On Jan 6, 2015, at 12:58 AM, Rahul = Bhardwaj <rahul.bhardwaj@indiamart.com> = wrote:

 

=

Hi,

 

We are using cassandra 2.1 version in a cluster of three machines each = with 64 GB RAM

 

The processes are killed by kernel, coz they are eating all memory = (oom-killer). We have set JAVA heap to default (i.e. it is using 8G) = because we have 64 GB RAM.

 

Please help.

 

 

Regards:

Rahul Bhardwaj




Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how Irrfan Khan gets his work done in no time on = IndiaMART, kyunki Kaam Yahin Banta = Hai!!!

 

 

 



Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how Irrfan Khan gets his work done in no time on = IndiaMART, kyunki Kaam Yahin Banta = Hai!!!

 

<= p class=3DMsoNormal> 

 



Follow IndiaMART.com for latest updates on this and more: Mobile Channel:

Watch how Irrfan Khan gets his work done in no time on = IndiaMART, kyunki Kaam Yahin Banta = Hai!!!

 

------=_NextPart_000_0499_01D029C5.09E122F0--