Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of michael.m.morris@gmail.com
 designates 209.85.214.44 as permitted sender)
MIME-Version: 1.0
Date: Thu, 16 Aug 2012 09:56:15 -0500
Message-ID: 
 <CAF8LmBHS5rHN9WjWQS5WVv1qzLMFr62OjxjkywqsjRACXGLsZQ@mail.gmail.com>
Subject: nodetool repair uses insane amount of disk space
From: Michael Morris <michael.m.morris@gmail.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=0015175d0406c8a99f04c7633e4e

--0015175d0406c8a99f04c7633e4e
Content-Type: text/plain; charset=ISO-8859-1

Occasionally as I'm doing my regular anti-entropy repair I end up with a
node that uses an exceptional amount of disk space (node should have about
5-6 GB of data on it, but ends up with 25+GB, and consumes the limited
amount of disk space I have available)

How come a node would consume 5x its normal data size during the repair
process?

My setup is kind of strange in that it's only about 80-100GB of data on a
35 node cluster, with 2 data centers and 3 racks, however the rack
assignments are unbalanced.  One data center has 8 nodes, and the other
data center is split into 2 racks with one rack of 9 nodes, and the other
with 18 nodes.  However, within each rack, the tokens are distributed
equally. It's a long sad story about how we ended up this way, but it
basically boils down to having to utilize existing resources to resolve a
production issue.

Additionally, the repair process takes (what I feel is) an extremely long
time to complete (36+ hours), and it always seems that nodes are streaming
data to each other, even on back-to-back executions of the repair.

Any help on these issues is appreciated.

- Mike

--0015175d0406c8a99f04c7633e4e
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Occasionally as I&#39;m doing my regular anti-entropy repair I end up with =
a node that uses an exceptional amount of disk space (node should have abou=
t 5-6 GB of data on it, but ends up with 25+GB, and consumes the limited am=
ount of disk space I have available)<br>

<br>How come a node would consume 5x its normal data size during the repair=
 process?<br><br>My setup is kind of strange in that it&#39;s only about 80=
-100GB of data on a 35 node cluster, with 2 data centers and 3 racks, howev=
er the rack assignments are unbalanced.=A0 One data center has 8 nodes, and=
 the other data center is split into 2 racks with one rack of 9 nodes, and =
the other with 18 nodes.=A0 However, within each rack, the tokens are distr=
ibuted equally. It&#39;s a long sad story about how we ended up this way, b=
ut it basically boils down to having to utilize existing resources to resol=
ve a production issue.<br>

<br>Additionally, the repair process takes (what I feel is) an extremely lo=
ng time to complete (36+ hours), and it always seems that nodes are streami=
ng data to each other, even on back-to-back executions of the repair.<br>
<br>Any help on these issues is appreciated.<br><br>- Mike<br><br>

--0015175d0406c8a99f04c7633e4e--