cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Job Thomas" <>
Subject RE: Reading SSTables Potential File Descriptor Leak 1.2.18
Date Wed, 24 Sep 2014 04:41:23 GMT
Hi, It look like the offset in keycache is wrong !!.
 refresh the keycache may solve the issue.
Thanks & Regards
Job M Thomas 
Platform & Technology


From: Tim Heckman []
Sent: Wed 9/24/2014 6:17 AM
Subject: Reading SSTables Potential File Descriptor Leak 1.2.18


I ran in to a problem today where Cassandra 1.2.18 exhausted its number of permitted open
file descriptors (65,535). This node has 256 tokens (vnodes) and runs in a test environment
with relatively little traffic/data.

As best I could tell, the majority of the file descriptors open were for a single SSTable
'.db' file. Looking in the error logs I found quite a few exceptions that looked to have been

ERROR [ReadStage:3817] 2014-09-19 07:00:11,056 (line 191) Exception in
thread Thread[ReadStage:3817,5,main]
java.lang.RuntimeException: java.lang.IllegalArgumentException: unable to seek to position
29049 in /mnt/var/lib/cassandra/data/path/to/file.db (1855 bytes) in read-only mode

Upon further investigation, it turns out this file became 'read-only' after the Cassandra
node was gracefully restarted last week. I'd imagine this is a discussion for another email

I fixed the issue by running:

nodetool scrub <Keyspace>
nodetool repair <Keyspace>

Attached to this email is one of the log entries/stacktrace for this exception.

Before opening a JIRA ticket I thought I'd reach out to the list to see if anyone has seen
any similar behavior as well as do a bit of source-diving to try and verify that the descriptor
is actually leaking.


View raw message