incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ben Chobot <be...@instructure.com>
Subject Re: lots of extra bytes on disk
Date Thu, 28 Mar 2013 18:03:04 GMT
Actually, due to a misconfiguration, we weren't snapshotting at all on some of the nodes that
are experiencing this problem. So while we've fixed that, snapshot don't explain the problem.

On Mar 28, 2013, at 10:54 AM, Hiller, Dean wrote:

> Have you cleaned up your snapshotsÅ those take extra space and don't just
> go away unless you delete them.
> 
> Dean
> 
> On 3/28/13 11:46 AM, "Ben Chobot" <bench@instructure.com> wrote:
> 
>> Are you also running 1.1.5? I'm wondering (ok hoping) that this might be
>> fixed if I upgrade.
>> 
>> On Mar 28, 2013, at 8:53 AM, Lanny Ripple wrote:
>> 
>>> We occasionally (twice now on a 40 node cluster over the last 6-8
>>> months) see this.  My best guess is that Cassandra can fail to mark an
>>> SSTable for cleanup somehow.  Forced GC's or reboots don't clear them
>>> out.  We disable thrift and gossip; drain; snapshot; shutdown; clear
>>> data/Keyspace/Table/*.db and restore (hard-linking back into place to
>>> avoid data transfer) from the just created snapshot; restart.
>>> 
>>> 
>>> On Mar 28, 2013, at 10:12 AM, Ben Chobot <bench@instructure.com> wrote:
>>> 
>>>> Some of my cassandra nodes in my 1.1.5 cluster show a large
>>>> discrepancy between what cassandra says the SSTables should sum up to,
>>>> and what df and du claim exist. During repairs, this is almost always
>>>> pretty bad, but post-repair compactions tend to bring those numbers to
>>>> within a few percent of each other... usually. Sometimes they remain
>>>> much further apart after compactions have finished - for instance, I'm
>>>> looking at one node now that claims to have 205GB of SSTables, but
>>>> actually has 450GB of files living in that CF's data directory. No
>>>> pending compactions, and the most recent compaction for this CF
>>>> finished just a few hours ago.
>>>> 
>>>> nodetool cleanup has no effect.
>>>> 
>>>> What could be causing these extra bytes, and how to get them to go
>>>> away? I'm ok with a few extra GB of unexplained data, but an extra
>>>> 245GB (more than all the data this node is supposed to have!) is a
>>>> little extreme.
>>> 
>> 
> 


Mime
View raw message