cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Carlos Scheidecker (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10861) Memory leak with Cassadra java driver 3.0.0-beta1 and Cassandra 3.0.1
Date Mon, 14 Dec 2015 21:57:46 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15056787#comment-15056787
] 

Carlos Scheidecker commented on CASSANDRA-10861:
------------------------------------------------

Michael,

Sure, sorry if I had tried to be brief.

I cannot reproduce this. It only occurs after 1 or 2 days. But it occurs. Once Tomcat is restarted
it is all good. As the API has not changed only the server and driver, the only errors I have
are on the application driver side.

There are no errors on Cassandra side that I could find either on system.log nor debug.log.
This seems to be a driver error.

The only thing I had from the same time on Cassandra debug.log is as follows:

INFO  [IndexSummaryManager:1] 2015-12-14 11:49:18,904 IndexSummaryManager.java:257 - Redistributing
index summaries
DEBUG [ScheduledTasks:1] 2015-12-14 12:49:15,198 ColumnFamilyStore.java:829 - Enqueuing flush
of size_estimates: 5545698 (1%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:141] 2015-12-14 12:49:15,198 Memtable.java:363 - Writing Memtable-size_estimates@1082810684(514.892KiB
serialized bytes, 52200 ops, 1%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:141] 2015-12-14 12:49:15,226 Memtable.java:396 - Completed flushing
/var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/ma-486-big-Data.db
(351.674KiB) for commitlog position ReplayPosition(segmentId=1449870554433, position=15986025)
DEBUG [ScheduledTasks:1] 2015-12-14 12:49:17,758 ColumnFamilyStore.java:829 - Enqueuing flush
of sstable_activity: 9294 (0%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:142] 2015-12-14 12:49:17,758 Memtable.java:363 - Writing Memtable-sstable_activity@1052341053(1.162KiB
serialized bytes, 168 ops, 0%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:142] 2015-12-14 12:49:17,770 Memtable.java:396 - Completed flushing
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-484-big-Data.db
(1.297KiB) for commitlog position ReplayPosition(segmentId=1449870554433, position=15986025)
DEBUG [CompactionExecutor:3412] 2015-12-14 12:49:17,814 CompactionTask.java:146 - Compacting
(c2e5d160-a29b-11e5-bbad-cd29e2a8cd4b) [/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-484-big-Data.db:level=0,
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-483-big-Data.db:level=0,
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-482-big-Data.db:level=0,
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-481-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:3412] 2015-12-14 12:49:17,902 CompactionTask.java:217 - Compacted
(c2e5d160-a29b-11e5-bbad-cd29e2a8cd4b) 4 sstables to [/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-485-big,]
to level=0.  2,555 bytes to 603 (~23% of original) in 88ms = 0.006535MB/s.  0 total partitions
merged to 14.  Partition merge counts were {1:8, 4:14, }
INFO  [IndexSummaryManager:1] 2015-12-14 12:49:18,914 IndexSummaryManager.java:257 - Redistributing
index summaries
DEBUG [ScheduledTasks:1] 2015-12-14 13:49:15,200 ColumnFamilyStore.java:829 - Enqueuing flush
of size_estimates: 5545698 (1%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:15,200 Memtable.java:363 - Writing Memtable-size_estimates@647480674(514.892KiB
serialized bytes, 52200 ops, 1%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:15,220 Memtable.java:396 - Completed flushing
/var/lib/cassandra/data/system/size_estimates-618f817b005f3678b8a453f3930b8e86/ma-487-big-Data.db
(351.674KiB) for commitlog position ReplayPosition(segmentId=1449870554433, position=20068341)
DEBUG [ScheduledTasks:1] 2015-12-14 13:49:17,756 ColumnFamilyStore.java:829 - Enqueuing flush
of compaction_history: 1147 (0%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:144] 2015-12-14 13:49:17,756 Memtable.java:363 - Writing Memtable-compaction_history@953834276(0.226KiB
serialized bytes, 1 ops, 0%/0% of on/off-heap limit)
DEBUG [ScheduledTasks:1] 2015-12-14 13:49:17,760 ColumnFamilyStore.java:829 - Enqueuing flush
of sstable_activity: 9902 (0%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:17,760 Memtable.java:363 - Writing Memtable-sstable_activity@481319531(1.193KiB
serialized bytes, 172 ops, 0%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:144] 2015-12-14 13:49:17,772 Memtable.java:396 - Completed flushing
/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-208-big-Data.db
(0.122KiB) for commitlog position ReplayPosition(segmentId=1449870554433, position=20068341)
DEBUG [MemtableFlushWriter:143] 2015-12-14 13:49:17,772 Memtable.java:396 - Completed flushing
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/ma-486-big-Data.db
(1.506KiB) for commitlog position ReplayPosition(segmentId=1449870554433, position=20068341)
DEBUG [CompactionExecutor:3460] 2015-12-14 13:49:17,860 CompactionTask.java:146 - Compacting
(24b13e40-a2a4-11e5-bbad-cd29e2a8cd4b) [/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-206-big-Data.db:level=0,
/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-207-big-Data.db:level=0,
/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-205-big-Data.db:level=0,
/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-208-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:3460] 2015-12-14 13:49:17,940 CompactionTask.java:217 - Compacted
(24b13e40-a2a4-11e5-bbad-cd29e2a8cd4b) 4 sstables to [/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/ma-209-big,]
to level=0.  6,021 bytes to 5,669 (~94% of original) in 80ms = 0.067580MB/s.  0 total partitions
merged to 123.  Partition merge counts were {1:126, }
INFO  [IndexSummaryManager:1] 2015-12-14 13:49:18,920 IndexSummaryManager.java:257 - Redistributing
index summaries

Cassandra state was just fine when those occurred. Never seemed to be on the server side but
on the driver side instead.

On my @Repository class here is the connect() method:

private void connect() {
		if (session == null) {
			String contactpoints = env.getProperty("cassandra.contactpoints");
			PlainTextAuthProvider plainAuthProvider = new PlainTextAuthProvider(
					env.getProperty("cassandra.username"),
					env.getProperty("cassandra.password"));
			// Probably needs to set Pooling Options etc
			// Cluster.builder().getConfiguration().getPoolingOptions()
			// Cluster.builder().withReconnectionPolicy(policy)
			// Cluster.builder().withRetryPolicy(policy)
			cluster = Cluster.builder()
					.addContactPoints(contactpoints.split(","))
					.withAuthProvider(plainAuthProvider).build();
			// connect() without keyspace
			// Session cs = cluster.connect();
			keyspace = env.getProperty("cassandra.oauthKeyspace");
			//session = cluster.connect(keyspace);
			// conect without specifying keyspace
			session = cluster.connect();
		}
	}

For the node configuration in terms of connections I have the default which is unlimited:

# The maximum number of concurrent client connections.
# The default is -1, which means unlimited.
# native_transport_max_concurrent_connections: -1

# The maximum number of concurrent client connections per source ip.
# The default is -1, which means unlimited.
# native_transport_max_concurrent_connections_per_ip: -1

I had also checked Tomcat limits and I have no limits on opened files with it:

cat /proc/30948/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             256854               256854               processes 
Max open files            4096                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       256854               256854               signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us    

> Memory leak with Cassadra java driver 3.0.0-beta1 and Cassandra 3.0.1
> ---------------------------------------------------------------------
>
>                 Key: CASSANDRA-10861
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10861
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Streaming and Messaging
>         Environment: Ubuntu 14.04.3 LTS 64 bits, Java build 1.8.0_66-b17, tomcat 8.0.23
>            Reporter: Carlos Scheidecker
>             Fix For: 3.0.1
>
>         Attachments: error_log_tomcat.txt
>
>
> Same dev environment with same application on Tomcat 8.0.23. However the dev nodes have
been upgraded to 3.0.0 and later to 3.0.1. The Cassandra driver is version 3.0.0-beta1.
> It seems that connections crash, do not get cleared and it leads to a memory leak stack
overflow condition.
> Attached is an error log file from tomcat.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message