Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy includes SPF record at
 spf.trusted-forwarder.org)
MIME-Version: 1.0
In-Reply-To: 
 <CAEDUwd1PtLMeG2pJiWYePH+4wy8LYhCAq9_vxGUR-66XQTfz-g@mail.gmail.com>
References: 
 <CAAzw3wstwxY90bnwOPzXqJokMomapjPS5Sac8F3vLG+2UsEOJQ@mail.gmail.com>
	<CAAzw3wsDbeAeNZA_26O7qoYSMpLSNtt6LvJt=U6YHBU_P=jQsg@mail.gmail.com>
	<CAJMN9iNUA1zrpUUHU_4xMzhTFom--t=-MZpcqoRJ2uhsWbhxew@mail.gmail.com>
	<CAAzw3wuAHgj48FQg6YBkogd8GMJzWX32h3iiwiv8ieX=uOf3Yw@mail.gmail.com>
	<CAEDUwd1PtLMeG2pJiWYePH+4wy8LYhCAq9_vxGUR-66XQTfz-g@mail.gmail.com>
Date: Thu, 12 Dec 2013 13:10:00 +1100
Message-ID: 
 <CAAzw3wsk=E8LVCg8DembbH6CMJRrmgX4VLW80xQOyGom_OLFtQ@mail.gmail.com>
Subject: Re: nodetool repair keeping an empty cluster busy
From: Sven Stark <sven.stark@m-square.com.au>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=001a11c2679ccd900c04ed4cd7a6

--001a11c2679ccd900c04ed4cd7a6
Content-Type: text/plain; charset=ISO-8859-1

Thanks Rob.

Well, I ran the repair only against the empty keyspace. C* version is
1.2.8. I guess I'll try to recreate it some time next week on two or three
test hosts and if the same behaviour occurs I'll file a bug report.

Cheers,
Sven


On Thu, Dec 12, 2013 at 12:58 PM, Robert Coli <rcoli@eventbrite.com> wrote:

> On Wed, Dec 11, 2013 at 1:35 AM, Sven Stark <sven.stark@m-square.com.au>wrote:
>
>> thanks for replying. Could you please be a bit more specific, though. Eg
>> what exactly is being compacted - there is/was no data at all in the
>> cluster save for a few hundred kB in the system CF (see the nodetool status
>> output). Or - how can those few hundred kB in data generate Gb of network
>> traffic?
>>
>
> The only answer I can come up with is that the Merkle trees generated and
> compared by repair are of a fixed size, and don't scale with the data
> present in the cluster. While I'm pretty sure each node can be aware that
> it has little to no data to repair, it generates and compares the trees
> anyway. It's a bit surprising that this might be Gbs of network traffic...
>
> The system keyspace will always have some data in it, have you tried only
> compacting your empty keyspace instead of the whole node?
>
> If so, and it exhibits the same behavior, that seems like a bug or at
> least unexpected behavior to me. If you're running a modern version of
> Cassandra, I would file a JIRA.
>
> =Rob
>
>

--001a11c2679ccd900c04ed4cd7a6
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Thanks Rob.<div><br></div><div>Well, I ran the repair only=
 against the empty keyspace. C* version is 1.2.8. I guess I&#39;ll try to r=
ecreate it some time next week on two or three test hosts and if the same b=
ehaviour occurs I&#39;ll file a bug report.</div>
<div><br></div><div>Cheers,</div><div>Sven</div></div><div class=3D"gmail_e=
xtra"><br><br><div class=3D"gmail_quote">On Thu, Dec 12, 2013 at 12:58 PM, =
Robert Coli <span dir=3D"ltr">&lt;<a href=3D"mailto:rcoli@eventbrite.com" t=
arget=3D"_blank">rcoli@eventbrite.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div class=3D"im">On Wed, D=
ec 11, 2013 at 1:35 AM, Sven Stark <span dir=3D"ltr">&lt;<a href=3D"mailto:=
sven.stark@m-square.com.au" target=3D"_blank">sven.stark@m-square.com.au</a=
>&gt;</span> wrote:<br>
</div><div class=3D"gmail_extra"><div class=3D"gmail_quote"><div class=3D"i=
m">
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr"><div>thanks for replying. C=
ould you please be a bit more specific, though. Eg what exactly is being co=
mpacted - there is/was no data at all in the cluster save for a few hundred=
 kB in the system CF (see the nodetool status output). Or - how can those f=
ew hundred kB in data generate Gb of network traffic?</div>

</div></blockquote><div><br></div></div><div>The only answer I can come up =
with is that the Merkle trees generated and compared by repair are of a fix=
ed size, and don&#39;t scale with the data present in the cluster. While I&=
#39;m pretty sure each node can be aware that it has little to no data to r=
epair, it generates and compares the trees anyway. It&#39;s a bit surprisin=
g that this might be Gbs of network traffic...</div>

<div><br></div><div>The system keyspace will always have some data in it, h=
ave you tried only compacting your empty keyspace instead of the whole node=
?</div><div><br></div><div>If so, and it exhibits the same behavior, that s=
eems like a bug or at least unexpected behavior to me. If you&#39;re runnin=
g a modern version of Cassandra, I would file a JIRA.</div>

<div><br></div><div>=3DRob</div><div><br></div></div></div></div>
</blockquote></div><br></div>

--001a11c2679ccd900c04ed4cd7a6--