cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonas Borgström (Commented) (JIRA) <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-3616) Temp SSTable and file descriptor leak
Date Wed, 14 Dec 2011 16:57:30 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-3616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13169498#comment-13169498
] 

Jonas Borgström commented on CASSANDRA-3616:
--------------------------------------------

Just a quick "me too".

After upgrading from 1.0.3 to 1.0.6 I see a bunch of left over 0 byte tmp files:
{code}
-rw-r--r--  1 cassandra 215          0 Dec 14 16:25 LogTokenIndex-tmp-hc-3175-Data.db
-rw-r--r--  1 cassandra 215          0 Dec 14 16:25 LogTokenIndex-tmp-hc-3175-Index.db
-rw-r--r--  1 cassandra 215          0 Dec 14 16:33 LogToken-tmp-hc-3093-Data.db
-rw-r--r--  1 cassandra 215          0 Dec 14 16:33 LogToken-tmp-hc-3093-Index.db
-rw-r--r--  1 cassandra 215          0 Dec 14 16:37 LogToken-tmp-hc-3104-Data.db
-rw-r--r--  1 cassandra 215          0 Dec 14 16:37 LogToken-tmp-hc-3104-Index.db
-rw-r--r--  1 cassandra 215          0 Dec 14 16:37 LogToken-tmp-hc-3105-Data.db
-rw-r--r--  1 cassandra 215          0 Dec 14 16:37 LogToken-tmp-hc-3105-Index.db
{code}
The timestamp suggests that they were created during a nodetool repair session.
                
> Temp SSTable and file descriptor leak
> -------------------------------------
>
>                 Key: CASSANDRA-3616
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-3616
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.0.5
>         Environment: 1.0.5 + CASSANDRA-3532 patch
> Solaris 10
>            Reporter: Eric Parusel
>
> Discussion about this started in CASSANDRA-3532.  It's on it's own ticket now.
> Anyhow:
> The nodes in my cluster are using a lot of file descriptors, holding open tmp files.
A few are using 50K+, nearing their limit (on Solaris, of 64K).
> Here's a small snippet of lsof:
> java 828 appdeployer *162u VREG 181,65540 0 333884 /data1/cassandra/data/MA_DDR/messages_meta-tmp-hb-776518-Data.db
> java 828 appdeployer *163u VREG 181,65540 0 333502 /data1/cassandra/data/MA_DDR/messages_meta-tmp-hb-776452-Data.db
> java 828 appdeployer *165u VREG 181,65540 0 333929 /data1/cassandra/data/MA_DDR/messages_meta-tmp-hb-776527-Index.db
> java 828 appdeployer *166u VREG 181,65540 0 333859 /data1/cassandra/data/MA_DDR/messages_meta-tmp-hb-776514-Data.db
> java 828 appdeployer *167u VREG 181,65540 0 333663 /data1/cassandra/data/MA_DDR/messages_meta-tmp-hb-776480-Data.db
> java 828 appdeployer *168u VREG 181,65540 0 333812 /data1/cassandra/data/MA_DDR/messages_meta-tmp-hb-776506-Index.db
> I spot checked a few and found they still exist on the filesystem too:
> rw-rr- 1 appdeployer appdeployer 0 Dec 12 07:16 /data1/cassandra/data/MA_DDR/messages_meta-tmp-hb-776506-Index.db
> After more investigation, it seems to happen during a CompactionTask.
> I waited until I saw some -tmp- files hanging around in the data dir:
> -rw-r--r--   1 appdeployer appdeployer       0 Dec 12 21:47:10 2011 messages_meta-tmp-hb-788904-Data.db
> -rw-r--r--   1 appdeployer appdeployer       0 Dec 12 21:47:10 2011 messages_meta-tmp-hb-788904-Index.db
> and then found this in the logs:
>  INFO [CompactionExecutor:18839] 2011-12-12 21:47:07,173 CompactionTask.java (line 113)
Compacting [SSTableReader(path='/data1/cassandra/data/MA_DDR/messages_meta-hb-760408-Data.db'),
SSTableReader(path='/data1/cassandra/data/MA_DDR/messages_meta-hb-760413-Data.db'), SSTableReader(path='/data1/cassandra/data/MA_DDR/messages_meta-hb-760409-Data.db'),
SSTableReader(path='/data1/cassandra/data/MA_DDR/messages_meta-hb-788314-Data.db'), SSTableReader(path='/data1/cassandra/data/MA_DDR/messages_meta-hb-760407-Data.db'),
SSTableReader(path='/data1/cassandra/data/MA_DDR/messages_meta-hb-760412-Data.db'), SSTableReader(path='/data1/cassandra/data/MA_DDR/messages_meta-hb-760410-Data.db'),
SSTableReader(path='/data1/cassandra/data/MA_DDR/messages_meta-hb-760411-Data.db')]
> INFO [CompactionExecutor:18839] 2011-12-12 21:47:10,461 CompactionTask.java (line 218)
Compacted to [/data1/cassandra/data/MA_DDR/messages_meta-hb-788896-Data.db,/data1/cassandra/data/MA_DDR/messages_meta-hb-788897-Data.db,/data1/cassandra/data/MA_DDR/messages_meta-hb-788898-Data.db,/data1/cassandra/data/MA_DDR/messages_meta-hb-788899-Data.db,/data1/cassandra/data/MA_DDR/messages_meta-hb-788900-Data.db,/data1/cassandra/data/MA_DDR/messages_meta-hb-788901-Data.db,/data1/cassandra/data/MA_DDR/messages_meta-hb-788902-Data.db,/data1/cassandra/data/MA_DDR/messages_meta-hb-788903-Data.db,].
 83,899,295 to 83,891,657 (~99% of original) bytes for 75,662 keys at 24.332518MB/s.  Time:
3,288ms.
> Note that the timestamp of the 2nd log line matches the last modified time of the files,
and has IDs leading up to, *but not including 788904*.
> I thought this might be relavent information, but I haven't found the specific cause
yet.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message