cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From 李建奇 <lijia...@jd.com>
Subject 答复: Cassandra consuming whole RAM (64 G)
Date Tue, 06 Jan 2015 07:25:44 GMT
Hi ,

     There is a “possible memory leak “ issue with c* 2.1.2.   https://issues.apache.org/jira/browse/CASSANDRA-8248

It happen with our c* 2.1.2 cluster .   In /proc/{pid}/maps there are a lot of deleted file
maps .

   

 

  _____  

Stephen li 

 

 

发件人: Joe Ramsey [mailto:joe.ramsey@mac.com] 
发送时间: 2015年1月6日 15:12
收件人: user@cassandra.apache.org
主题: Re: Cassandra consuming whole RAM (64 G)

 

Thanks Rahul and good luck!  I’m really curious to hear what the result is.

 

 

On Jan 6, 2015, at 2:10 AM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com> wrote:

 

Thanks for your response.. i will get back to you with my findings.

 

On Tue, Jan 6, 2015 at 12:36 PM, Joe Ramsey <joe.ramsey@mac.com> wrote:

That should be “writing too many bytes” not “waiting too many bytes” just for clarity’s
sake.

 

On Jan 6, 2015, at 2:03 AM, Joe Ramsey <joe.ramsey@mac.com> wrote:

 

I’m not an expert.  Really just learning this myself but it looks like according to the
stack you’re getting an exception waiting too many bytes to the commit log.  

 

That’s controlled by commit_log_segment_size_in_mb setting.  The maximum write size that
C* will allow is half of the value set for this parameter so if it’s set for 32 (default)
that means your max write would be 16MB.  (And that’s what’s getting reported in the stack
trace.  You’re writing 16965030 bytes (16.18M) and your max write is 16MB so it’s throwing
the exception.  Try setting commit_log_segment_size_in_mb from 32 to 64 and see if the error
goes away.  It really should get tuned for the amount of writes but that’ll tell you at
least if this is the issue.

 

Let me know how it goes!

 

Joe

 

 

 

On Jan 6, 2015, at 1:40 AM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com> wrote:

 

Hi Joe,

 

PFB output of system.log

 

 tail -n 100 system.log

INFO  [CompactionExecutor:164] 2015-01-06 11:58:28,555 CompactionTask.java:251 - Compacted
4 sstables to [/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-243,].
 50,514,079 bytes to 50,626,490 (~100% of original) in 654,529ms = 0.073765MB/s.  277,719
total partitions merged to 260,285.  Partition merge counts were {1:246186, 2:11302, 3:2259,
4:538, }

INFO  [CompactionExecutor:176] 2015-01-06 11:58:28,579 CompactionTask.java:251 - Compacted
4 sstables to [/var/lib/cassandra/data/system/compactions_in_progress-55080ab05d9c388690a4acb25fe1f77b/system-compactions_in_progress-ka-882,].
 511 bytes to 42 (~8% of original) in 27ms = 0.001483MB/s.  7 total partitions merged to 1.
 Partition merge counts were {1:1, 2:3, }

WARN  [Thrift:16] 2015-01-06 11:59:02,351 BatchStatement.java:255 - Batch of prepared statements
for [clickstream.business_feed] is of size 433053, exceeding specified threshold of 5120 by
427933.

INFO  [CompactionExecutor:175] 2015-01-06 11:59:19,883 CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5646-Data.db')]

INFO  [CompactionExecutor:175] 2015-01-06 11:59:19,952 CompactionTask.java:251 - Compacted
1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647,].
 2,801,126 bytes to 2,801,126 (~100% of original) in 67ms = 39.871073MB/s.  26 total partitions
merged to 26.  Partition merge counts were {1:26, }

WARN  [Thrift:4] 2015-01-06 12:00:02,457 BatchStatement.java:255 - Batch of prepared statements
for [clickstream.business_feed] is of size 433819, exceeding specified threshold of 5120 by
428699.

INFO  [CompactionExecutor:178] 2015-01-06 12:00:19,982 CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5647-Data.db')]

INFO  [CompactionExecutor:178] 2015-01-06 12:00:20,048 CompactionTask.java:251 - Compacted
1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648,].
 2,801,126 bytes to 2,801,126 (~100% of original) in 65ms = 41.097876MB/s.  26 total partitions
merged to 26.  Partition merge counts were {1:26, }

WARN  [Thrift:5] 2015-01-06 12:01:02,387 BatchStatement.java:255 - Batch of prepared statements
for [clickstream.business_feed] is of size 436755, exceeding specified threshold of 5120 by
431635.

INFO  [CompactionExecutor:179] 2015-01-06 12:01:20,074 CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5648-Data.db')]

INFO  [CompactionExecutor:179] 2015-01-06 12:01:20,139 CompactionTask.java:251 - Compacted
1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649,].
 2,801,126 bytes to 2,801,126 (~100% of original) in 63ms = 42.402570MB/s.  26 total partitions
merged to 26.  Partition merge counts were {1:26, }

WARN  [SharedPool-Worker-1] 2015-01-06 12:01:29,692 AbstractTracingAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}

java.lang.IllegalArgumentException: Mutation of 17225046 bytes is too large for the maxiumum
size of 16777216

        at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]

        at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]

INFO  [CompactionExecutor:180] 2015-01-06 12:02:20,167 CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5649-Data.db')]

INFO  [CompactionExecutor:180] 2015-01-06 12:02:20,241 CompactionTask.java:251 - Compacted
1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650,].
 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.  26 total partitions
merged to 26.  Partition merge counts were {1:26, }

INFO  [Service Thread] 2015-01-06 12:03:02,108 GCInspector.java:142 - ParNew GC in 380ms.
 CMS Old Gen: 1551865712 -> 1704414952; Par Eden Space: 1718091776 -> 0; Par Survivor
Space: 161099424 -> 214695936

WARN  [Thrift:4] 2015-01-06 12:03:03,357 BatchStatement.java:255 - Batch of prepared statements
for [clickstream.im_csl_glusr_daily_activity_count] is of size 60021, exceeding specified
threshold of 5120 by 54901.

INFO  [CompactionExecutor:181] 2015-01-06 12:03:20,274 CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5650-Data.db')]

INFO  [CompactionExecutor:181] 2015-01-06 12:03:20,348 CompactionTask.java:251 - Compacted
1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651,].
 2,801,126 bytes to 2,801,126 (~100% of original) in 73ms = 36.593999MB/s.  26 total partitions
merged to 26.  Partition merge counts were {1:26, }

WARN  [Thrift:3] 2015-01-06 12:04:02,524 BatchStatement.java:255 - Batch of prepared statements
for [clickstream.business_feed] is of size 517769, exceeding specified threshold of 5120 by
512649.

INFO  [CompactionExecutor:182] 2015-01-06 12:04:20,374 CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5651-Data.db')]

INFO  [CompactionExecutor:182] 2015-01-06 12:04:20,445 CompactionTask.java:251 - Compacted
1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652,].
 2,801,126 bytes to 2,801,126 (~100% of original) in 69ms = 38.715390MB/s.  26 total partitions
merged to 26.  Partition merge counts were {1:26, }

WARN  [Thrift:10] 2015-01-06 12:04:24,967 BatchStatement.java:255 - Batch of prepared statements
for [clickstream.im_csl_log] is of size 13220136, exceeding specified threshold of 5120 by
13215016.

WARN  [SharedPool-Worker-1] 2015-01-06 12:04:25,092 AbstractTracingAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}

java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 16800887 bytes
is too large for the maxiumum size of 16777216

        at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]

        at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]

Caused by: java.lang.IllegalArgumentException: Mutation of 16800887 bytes is too large for
the maxiumum size of 16777216

        at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        ... 4 common frames omitted

INFO  [CompactionExecutor:183] 2015-01-06 12:05:20,472 CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5652-Data.db')]

INFO  [CompactionExecutor:183] 2015-01-06 12:05:20,538 CompactionTask.java:251 - Compacted
1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653,].
 2,801,126 bytes to 2,801,126 (~100% of original) in 64ms = 41.740030MB/s.  26 total partitions
merged to 26.  Partition merge counts were {1:26, }

WARN  [Thrift:8] 2015-01-06 12:05:24,982 BatchStatement.java:255 - Batch of prepared statements
for [clickstream.im_csl_log] is of size 13136950, exceeding specified threshold of 5120 by
13131830.

INFO  [SlabPoolCleaner] 2015-01-06 12:05:26,129 ColumnFamilyStore.java:840 - Enqueuing flush
of im_csl_log: 505903980 (24%) on-heap, 0 (0%) off-heap

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:26,131 Memtable.java:325 - Writing Memtable-im_csl_log@1246880835(90297505
serialized bytes, 1530359 ops, 20%/0% of on/off-heap limit)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:28,623 Memtable.java:364 - Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log-ka-2654-Data.db
(21516225 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:28,625 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_modid_idx@118242201(2110840
serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:28,723 Memtable.java:364 - Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_modid_idx-ka-2710-Data.db
(361277 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:28,724 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx@823892581(2110840
serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:28,848 Memtable.java:364 - Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_email_idx-ka-2459-Data.db
(543861 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:28,850 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx@188053011(2110840
serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:28,988 Memtable.java:364 - Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_catalog_owner_glusr_id_idx-ka-245-Data.db
(481902 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:28,989 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_v4iilex_glusr_id_idx@1658511140(2110840
serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:364 - Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_v4iilex_glusr_id_idx-ka-2523-Data.db
(460053 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:29,103 Memtable.java:325 - Writing Memtable-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx@258991920(2110840
serialized bytes, 52771 ops, 1%/0% of on/off-heap limit)

INFO  [MemtableFlushWriter:44] 2015-01-06 12:05:29,211 Memtable.java:364 - Completed flushing
/var/lib/cassandra/data/clickstream/im_csl_log-22207f2081bb11e4abd4a9d4f1e0b940/clickstream-im_csl_log.im_csl_log_imeshvisitor_glusr_id_idx-ka-2831-Data.db
(395084 bytes) for commitlog position ReplayPosition(segmentId=1420516895898, position=16708232)

INFO  [CompactionExecutor:185] 2015-01-06 12:06:20,563 CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5653-Data.db')]

INFO  [CompactionExecutor:185] 2015-01-06 12:06:20,633 CompactionTask.java:251 - Compacted
1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654,].
 2,801,126 bytes to 2,801,126 (~100% of original) in 68ms = 39.284734MB/s.  26 total partitions
merged to 26.  Partition merge counts were {1:26, }

INFO  [CompactionExecutor:187] 2015-01-06 12:07:20,659 CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5654-Data.db')]

INFO  [CompactionExecutor:187] 2015-01-06 12:07:20,733 CompactionTask.java:251 - Compacted
1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655,].
 2,801,126 bytes to 2,801,126 (~100% of original) in 72ms = 37.102249MB/s.  26 total partitions
merged to 26.  Partition merge counts were {1:26, }

WARN  [Thrift:6] 2015-01-06 12:07:26,364 BatchStatement.java:255 - Batch of prepared statements
for [clickstream.im_csl_log] is of size 13480721, exceeding specified threshold of 5120 by
13475601.

WARN  [SharedPool-Worker-1] 2015-01-06 12:07:26,537 AbstractTracingAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}

java.lang.RuntimeException: java.lang.IllegalArgumentException: Mutation of 17138032 bytes
is too large for the maxiumum size of 16777216

        at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2121)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]

        at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]

Caused by: java.lang.IllegalArgumentException: Mutation of 17138032 bytes is too large for
the maxiumum size of 16777216

        at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.service.StorageProxy$7.runMayThrow(StorageProxy.java:999)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2117)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        ... 4 common frames omitted

INFO  [CompactionExecutor:188] 2015-01-06 12:08:20,762 CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5655-Data.db')]

INFO  [CompactionExecutor:188] 2015-01-06 12:08:20,825 CompactionTask.java:251 - Compacted
1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656,].
 2,801,126 bytes to 2,801,126 (~100% of original) in 60ms = 44.522699MB/s.  26 total partitions
merged to 26.  Partition merge counts were {1:26, }

INFO  [CompactionExecutor:189] 2015-01-06 12:09:20,853 CompactionTask.java:136 - Compacting
[SSTableReader(path='/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5656-Data.db')]

INFO  [CompactionExecutor:189] 2015-01-06 12:09:20,921 CompactionTask.java:251 - Compacted
1 sstables to [/var/lib/cassandra/data/system/batchlog-0290003c977e397cac3efdfdc01d626b/system-batchlog-ka-5657,].
 2,801,126 bytes to 2,801,126 (~100% of original) in 66ms = 40.475181MB/s.  26 total partitions
merged to 26.  Partition merge counts were {1:26, }

WARN  [SharedPool-Worker-1] 2015-01-06 12:09:22,532 AbstractTracingAwareExecutorService.java:169
- Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}

java.lang.IllegalArgumentException: Mutation of 16965030 bytes is too large for the maxiumum
size of 16777216

        at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:221) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:371) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:351) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:62) ~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[na:1.7.0_71]

        at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164)
~[apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.2.jar:2.1.2-SNAPSHOT]

        at java.lang.Thread.run(Unknown Source) [na:1.7.0_71]

 

 

 

Regards:

Rahul Bhardwaj

 

On Tue, Jan 6, 2015 at 12:08 PM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com> wrote:

Hi Joe,

 

PFA heap dump

 

 

regards:

Rahul Bhardwaj

 

 

 

On Tue, Jan 6, 2015 at 11:35 AM, Joe Ramsey <joe.ramsey@mac.com> wrote:

Did you try generating a heap dump so you can look through it to see what’s actually happened?

 

 

On Jan 6, 2015, at 12:58 AM, Rahul Bhardwaj <rahul.bhardwaj@indiamart.com> wrote:

 

Hi,

 

We are using cassandra 2.1 version in a cluster of three machines each with 64 GB RAM

 

The processes are killed by kernel, coz they are eating all memory (oom-killer). We have set
JAVA heap to default (i.e. it is using 8G) because we have 64 GB RAM.

 

Please help.

 

 

Regards:

Rahul Bhardwaj






Follow  <http://www.indiamart.com/> IndiaMART.com for latest updates on this and more:
 <https://plus.google.com/+indiamart>  <https://www.facebook.com/IndiaMART>  <https://twitter.com/IndiaMART>
Mobile Channel:  <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
 <https://play.google.com/store/apps/details?id=com.indiamart.m>  <http://m.indiamart.com/>


Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta
Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU> !!!

 

 

 



Follow  <http://www.indiamart.com/> IndiaMART.com for latest updates on this and more:
 <https://plus.google.com/+indiamart>  <https://www.facebook.com/IndiaMART>  <https://twitter.com/IndiaMART>
Mobile Channel:  <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
 <https://play.google.com/store/apps/details?id=com.indiamart.m>  <http://m.indiamart.com/>


Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta
Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU> !!!

 

 

 



Follow  <http://www.indiamart.com/> IndiaMART.com for latest updates on this and more:
 <https://plus.google.com/+indiamart>  <https://www.facebook.com/IndiaMART>  <https://twitter.com/IndiaMART>
Mobile Channel:  <https://itunes.apple.com/WebObjects/MZStore.woa/wa/viewSoftware?id=668561641&mt=8>
 <https://play.google.com/store/apps/details?id=com.indiamart.m>  <http://m.indiamart.com/>


Watch how Irrfan Khan gets his work done in no time on IndiaMART, kyunki Kaam Yahin Banta
Hai <https://www.youtube.com/watch?v=hmS4Afl2bNU> !!!

 


Mime
View raw message