Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; c=nofws; d=thelastpickle.com; h=content-type
	:mime-version:subject:from:in-reply-to:date
	:content-transfer-encoding:message-id:references:to; q=dns; s=
	thelastpickle.com; b=G9yrQtU++zJfm1jv/DFO7ta1apBb8ObSsn3UUEXQiYT
	q8MAUqbNUXO54Bwwl2k8jRWDNYj2Vo16vy7IAAzbfAEx39jtQCOGKt2hiUrcRaDr
	HvNqJmCZW1Y1waUOfwk7EuS9U6PxAXDX2Cb4Sp437BEo3dud0w7bnBoj2xsgE2JU
	=
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Apple Message framework v1244.3)
Subject: Re: 4/20 nodes get disproportionate amount of mutations
From: aaron morton <aaron@thelastpickle.com>
In-Reply-To: <549D0947-C093-4C7F-95A9-1786F84166B1@gmail.com>
Date: Tue, 23 Aug 2011 20:43:17 +1200
Content-Transfer-Encoding: quoted-printable
Message-Id: <76938F68-F558-424F-9079-3E32BA376360@thelastpickle.com>
References: <4701A184-927B-4F07-96A6-7919F73AA110@gmail.com>
 <CAO5xsd3No0CWhCj5sa5jfdaH8p187iUT3tq_6RPCA=zM29Lzbw@mail.gmail.com>
 <549D0947-C093-4C7F-95A9-1786F84166B1@gmail.com>
To: user@cassandra.apache.org

Dropped messages in ReadRepair is odd. Are you also dropping mutations ?=20=


There are two tasks performed on the ReadRepair stage. The digests are =
compared on this stage, and secondly the repair happens on the stage. =
Comparing digests is quick. Doing the repair could take a bit longer, =
all the cf's returned are collated, filtered and deletes removed. =20

We don't do background Read Repair on range scans, they do have =
foreground digest checking though.

What CL are you using ?=20

begin crazy theory:

	Could there be a very big row that is out of sync ? The =
increased RR would be resulting in mutations been sent back to the =
replicas. Which would give you a hot spot in mutations.
=09
	Check max compacted row size on the hot nodes.=20
=09
	Turn the logging up to DEBUG on the hot machines for =
o.a.c.service.RowRepairResolver and look for the "resolve:=85" message =
it has the time taken.

Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com

On 23/08/2011, at 7:52 PM, Jeremy Hanna wrote:

>=20
> On Aug 23, 2011, at 2:25 AM, Peter Schuller wrote:
>=20
>>> We've been having issues where as soon as we start doing heavy =
writes (via hadoop) recently, it really hammers 4 nodes out of 20.  =
We're using random partitioner and we've set the initial tokens for our =
20 nodes according to the general spacing formula, except for a few =
token offsets as we've replaced dead nodes.
>>=20
>> Is the hadoop job iterating over keys in the cluster in token order
>> perhaps, and you're generating writes to those keys? That would
>> explain a "moving hotspot" along the cluster.
>=20
> Yes - we're iterating over all the keys of particular column families, =
doing joins using pig as we enrich and perform measure calculations.  =
When we write, we're usually writing out for a certain small subset of =
keys which shouldn't have hotspots with RandomPartitioner afaict.
>=20
>>=20
>> --=20
>> / Peter Schuller (@scode on twitter)
>=20