cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: nodetool repair uses insane amount of disk space
Date Fri, 17 Aug 2012 08:40:54 GMT
I would take a look at the replication: whats the RF per DC and what does nodetool ring say.
It's hard (as in no recommended) to get NTS with rack allocation working correctly. Without
know much more I would try to understand what the topology is and if it can be simplified.


>> Additionally, the repair process takes (what I feel is) an extremely long time to
complete (36+ hours), and it always seems that nodes are streaming data to each other, even
on back-to-back executions of the repair.
Run some metrics to clock the network IO during repair. 
Also run an experiment to repair a single CF twice from the same node and look at the logs
for the second run. This will give us an idea of how much data is being transferred. 
Note that very wide rows can result in large repair transfers as the whole row is diff'd and
transferred if needed.
 
Hope that helps. 


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 17/08/2012, at 11:14 AM, Michael Morris <michael.m.morris@gmail.com> wrote:

> Upgraded to 1.1.3 from 1.0.8 about 2 weeks ago.
> 
> On Thu, Aug 16, 2012 at 5:57 PM, aaron morton <aaron@thelastpickle.com> wrote:
> What version are using ? There were issues with repair using lots-o-space in 0.8.X, it's
fixed in 1.X
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 17/08/2012, at 2:56 AM, Michael Morris <michael.m.morris@gmail.com> wrote:
> 
>> Occasionally as I'm doing my regular anti-entropy repair I end up with a node that
uses an exceptional amount of disk space (node should have about 5-6 GB of data on it, but
ends up with 25+GB, and consumes the limited amount of disk space I have available)
>> 
>> How come a node would consume 5x its normal data size during the repair process?
>> 
>> My setup is kind of strange in that it's only about 80-100GB of data on a 35 node
cluster, with 2 data centers and 3 racks, however the rack assignments are unbalanced.  One
data center has 8 nodes, and the other data center is split into 2 racks with one rack of
9 nodes, and the other with 18 nodes.  However, within each rack, the tokens are distributed
equally. It's a long sad story about how we ended up this way, but it basically boils down
to having to utilize existing resources to resolve a production issue.
>> 
>> Additionally, the repair process takes (what I feel is) an extremely long time to
complete (36+ hours), and it always seems that nodes are streaming data to each other, even
on back-to-back executions of the repair.
>> 
>> Any help on these issues is appreciated.
>> 
>> - Mike
>> 
> 
> 


Mime
View raw message