Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	thelastpickle.com; b=omHK3h8edgRf+fT08hTtHl4BeokYBM54B2k+8ZukrCB
	C4g/euZu0mVP5EFd6a6XkJ+DK8FKd6veKsFdPbuv0UYIAnnO7kd5OD409RWI0KjM
	ZeJLCZlgyNNc9gICZlO6o5QQjRQ/IwK8i45udUac9+o2Q3Wy+STDw3+UzV06RqjA
	=
Content-Type: text/plain; charset=iso-8859-1
Mime-Version: 1.0 (Apple Message framework v1082.1)
Subject: Re: AW: Strange nodetool repair behaviour
From: aaron morton <aaron@thelastpickle.com>
In-Reply-To: 
 <120CB7532EA53A4D8CA6B63F94B4ADB351E6D62840@IE2RD2XVS021.red002.local>
Date: Mon, 4 Apr 2011 22:46:32 +1000
Content-Transfer-Encoding: quoted-printable
Message-Id: <F14334C2-2EB3-46B9-8435-35A4E69C1280@thelastpickle.com>
References: <4D999CBB.4070603@trioptima.com>
 <120CB7532EA53A4D8CA6B63F94B4ADB351E6D62840@IE2RD2XVS021.red002.local>
To: user@cassandra.apache.org

Jonas, AFAIK if repair completed successfully there should be no =
streaming the next time round. This sounds odd please look into it if =
you can.=20

Can you run at DEBUG logging, there will be some messages about =
receiving streams from files and which ranges are being requested.=20

I would be interested to know if the repair is completing successfully. =
You should see messages such as "Repair session blah completed =
successfully"  if it is. It is possible repair to hang if one of the =
neighbours goes away or fails to send the data. In this case the repair =
session will timeout after 48 hours.=20

Aaron

On 4 Apr 2011, at 20:39, Roland Gude wrote:

> I am experiencing the same behavior but had it on previous versions of =
0.7 as well.
>=20
>=20
> -----Urspr=FCngliche Nachricht-----
> Von: Jonas Borgstr=F6m [mailto:jonas.borgstrom@trioptima.com]=20
> Gesendet: Montag, 4. April 2011 12:26
> An: user@cassandra.apache.org
> Betreff: Strange nodetool repair behaviour
>=20
> Hi,
>=20
> I have a 6 node 0.7.4 cluster with replication_factor=3D3 where =
"nodetool
> repair keyspace" behaves really strange.
>=20
> The keyspace contains three column families and about 60GB data in =
total
> (i.e 30GB on each node).
>=20
> Even though no data has been added or deleted since the last repair, a
> repair takes hours and the repairing node seems to receive 100+GB =
worth
> of sstable data from its neighbourhood nodes, i.e several times the
> actual data size.
>=20
> The log says things like:
>=20
> "Performing streaming repair of 27 ranges"
>=20
> And a bunch of:
>=20
> "Compacted to <filename> 22,208,983,964 to 4,816,514,033 (~21% of =
original)"
>=20
> In the end the repair finishes without any error after a few hours but
> even then the active sstables seems to contain lots of redundant data
> since the disk usage can be sliced in half by triggering a major =
compaction.
>=20
> All this leads me to believe that something stops the AES from =
correctly
> figuring out what data is already on the repairing node and what needs
> to be streamed from the neighbours.
>=20
> The only thing I can think of right now is that one of the column
> families contains a lot of large rows that are larger than
> memtable_throughput and that's perhaps what's confusing the merkle =
tree.
>=20
> Anyway, is this a known problem of perhaps expected behaviour?
> Otherwise I'll try to create a more reproducible test case.
>=20
> Regards,
> Jonas
>=20
>=20