incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jim Cistaro <jcist...@netflix.com>
Subject Re: nodetool repair uses insane amount of disk space
Date Fri, 17 Aug 2012 15:49:51 GMT
We see similar issues with some of the repairs at Netflix.

Regarding the growth in payload… we see similar symptoms where nodes can double or triple
size.  Part of this may be because the repair may deal in large chunks for comparisons.  This
means that even if there is one byte of entropy, you copy over a large chunk.  Another reason
for the large growth is that if node A is inconsistent with replicas on B and C, you will
copy over multiple sets of large chunks (one from each of the replicas) - even more sets in
a multi datacenter environment.  (We are still investigating/analyzing the causes of such
occurrences in our clusters – the above explanation is a possible cause.)

Are you only seeing growth on one node in the system?  You might want to check if other nodes
logs show gossip issues with this node (and then check if you are creating a lot of hints
and check your hint settings to make sure you save and replay them) and that may be why you
see this even on back to back execution.

It is worth noting that we do major compactions (I am not suggesting you do this, just pointing
it out for reference) and then see the payload shrink back down to normal.  So a lot of that
payload increase appears to be redundant (most likely due to the chunking issue above).

Regarding processing time… Are you repairing each node serially? Are you repairing with
primary range option?
AFAIK … You most likely want to use –pr.  Otherwise the further you get into the list
of nodes, the more data has to go through the validation compaction (because you increased
the size of some of your nodes).  Using –pr means you only repair a range once when repairing
the cluster.  Without it, you repair the range on each node/replica.

Jim


From: aaron morton <aaron@thelastpickle.com<mailto:aaron@thelastpickle.com>>
Reply-To: <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Date: Fri, 17 Aug 2012 20:40:54 +1200
To: <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: nodetool repair uses insane amount of disk space

I would take a look at the replication: whats the RF per DC and what does nodetool ring say.
It's hard (as in no recommended) to get NTS with rack allocation working correctly. Without
know much more I would try to understand what the topology is and if it can be simplified.

Additionally, the repair process takes (what I feel is) an extremely long time to complete
(36+ hours), and it always seems that nodes are streaming data to each other, even on back-to-back
executions of the repair.
Run some metrics to clock the network IO during repair.
Also run an experiment to repair a single CF twice from the same node and look at the logs
for the second run. This will give us an idea of how much data is being transferred.
Note that very wide rows can result in large repair transfers as the whole row is diff'd and
transferred if needed.

Hope that helps.


-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 17/08/2012, at 11:14 AM, Michael Morris <michael.m.morris@gmail.com<mailto:michael.m.morris@gmail.com>>
wrote:

Upgraded to 1.1.3 from 1.0.8 about 2 weeks ago.

On Thu, Aug 16, 2012 at 5:57 PM, aaron morton <aaron@thelastpickle.com<mailto:aaron@thelastpickle.com>>
wrote:
What version are using ? There were issues with repair using lots-o-space in 0.8.X, it's fixed
in 1.X

Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com<http://www.thelastpickle.com/>

On 17/08/2012, at 2:56 AM, Michael Morris <michael.m.morris@gmail.com<mailto:michael.m.morris@gmail.com>>
wrote:

Occasionally as I'm doing my regular anti-entropy repair I end up with a node that uses an
exceptional amount of disk space (node should have about 5-6 GB of data on it, but ends up
with 25+GB, and consumes the limited amount of disk space I have available)

How come a node would consume 5x its normal data size during the repair process?

My setup is kind of strange in that it's only about 80-100GB of data on a 35 node cluster,
with 2 data centers and 3 racks, however the rack assignments are unbalanced.  One data center
has 8 nodes, and the other data center is split into 2 racks with one rack of 9 nodes, and
the other with 18 nodes.  However, within each rack, the tokens are distributed equally. It's
a long sad story about how we ended up this way, but it basically boils down to having to
utilize existing resources to resolve a production issue.

Additionally, the repair process takes (what I feel is) an extremely long time to complete
(36+ hours), and it always seems that nodes are streaming data to each other, even on back-to-back
executions of the repair.

Any help on these issues is appreciated.

- Mike





Mime
View raw message