cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philippe <watche...@gmail.com>
Subject Re: nodetool repair caused high disk space usage
Date Wed, 17 Aug 2011 22:10:18 GMT
Huy,
Have you tried repairing one keyspace at a time and then giving it some
breathing time to compact.
My current observations is that the streams of repairs are triggering
massive compactions which are filling up my disks too. Another idea I'd like
to try is to limit the number of concurrent compactions because it appears
that "Validation" are not constrained by it. That way, I would have fewer
concurrent compactions are the disk would not fill up (as fast?)

2011/8/17 Huy Le <huyle@springpartners.com>

> I restarted the cluster and kicked off repair on the same node again.  It
> only made the matter worse.  It filled up the 830GB partition, and cassandra
> on the node repair ran on crashed.  I restarted it, and now I am running
> compaction to reduce disk usage.
>
> Repair after upgrading to 0.8.4 is still a problem.  Does anyone have any
> further info on the issue?  Thanks!
>
> Huy
>
>
> On Wed, Aug 17, 2011 at 11:13 AM, Huy Le <huyle@springpartners.com> wrote:
>
>> Sorry for the duplicate thread.  I saw the issue being referenced to
>> https://issues.apache.org/jira/browse/CASSANDRA-2280.   However, I am
>> running version 0.8.4.   I saw your comment in on of the threads that the
>> issue is not reprocible, but multiple users have the same issue.  This there
>> anything that I should do to determine the cause of this issue for I do a
>> rolling restart and try to run repair again?  Thanks!
>>
>> Huy
>>
>>
>> On Wed, Aug 17, 2011 at 11:03 AM, Philippe <watcherfr@gmail.com> wrote:
>>
>>> Look at my last two or three threads. I've encountered the same thing and
>>> got some pointers/answers.
>>> On Aug 17, 2011 4:03 PM, "Huy Le" <huyle@springpartners.com> wrote:
>>> > Hi,
>>> >
>>> > After upgrading to cass 0.8.4 from cass 0.6.11. I ran scrub. That
>>> worked
>>> > fine. Then I ran nodetool repair on one of the nodes. The disk usage on
>>> > data directory increased from 40GB to 480GB, and it's still growing.
>>> >
>>> > The cluster has 4 nodes with replica factor 3. The ring shows:
>>> >
>>> > Address DC Rack Status State Load
>>> > Owns Token
>>> >
>>> > 141784319550391026443072753096570088103
>>> > 10.245.xxx.xxx datacenter1 rack1 Up Normal 39.89 GB
>>> > 25.00% 14178431955039102644307275309657008807
>>> > 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.98 GB
>>> > 25.00% 56713727820156410577229101238628035239
>>> > 10.242.xxx.xxx datacenter1 rack1 Up Normal 80.08 GB
>>> > 25.00% 99249023685273718510150927167599061671
>>> > 10.244.xxx.xxx datacenter1 rack1 Up Normal 80.23 GB
>>> > 25.00% 141784319550391026443072753096570088103
>>> >
>>> > Does anyone know what's might be causing this issue?
>>> >
>>> > Huy
>>> >
>>> > --
>>> > Huy Le
>>> > Spring Partners, Inc.
>>> > http://springpadit.com
>>>
>>
>>
>>
>> --
>> Huy Le
>> Spring Partners, Inc.
>> http://springpadit.com
>>
>
>
>
> --
> Huy Le
> Spring Partners, Inc.
> http://springpadit.com
>

Mime
View raw message