cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Illarion Kovalchuk (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-5308) cassandra with leveled comaction quickly runs out of free space with tons of tmp files
Date Mon, 04 Mar 2013 14:29:13 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13592234#comment-13592234
] 

Illarion Kovalchuk commented on CASSANDRA-5308:
-----------------------------------------------

Yes, I expected the failures preventing compaction to finish to be there, but no - there were
no errors.
                
> cassandra with leveled comaction quickly runs out of free space with tons of tmp files
> --------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-5308
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5308
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.10
>         Environment: Cluster with 4 nodes running cassandra 1.1.10
>            Reporter: Illarion Kovalchuk
>             Fix For: 1.1.11
>
>         Attachments: cassandra-1.1.10-tmpfiles-symptoms.txt
>
>
> We were performing a massive upload of data to our cluster and found, that one of the
nodes is ruuning out of free space
> nodetool ring didn't show any extra usage, but df -h / on the problamatic node shown
us 95% of used space, while other nodes were all around 50%
> Eventually, we discovered that there's an abnormous peak of tmp files in one of our CF's:
753'906'319'257 bytes of data in 26811 files.
> Rebooting the node caused all that files to get deleted and node went back to normal.
> One of the symptoms: when the node started to misbehave and collect the temp files, it
started to flush the memtables due to lot of heap used.
> See attached info

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message