Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
Subject: Re: Read-repair working, repair not working?
From: aaron morton <aaron@thelastpickle.com>
In-Reply-To: 
 <CACPZYUE_NM_aUpu_JSLRtW6zjskVdvY9pF6wUau_2jnGvdvRTA@mail.gmail.com>
Date: Mon, 11 Feb 2013 09:12:59 +1300
Content-Transfer-Encoding: quoted-printable
Message-Id: <7869B1C1-E3C0-4E74-844F-D8CE56DFA219@thelastpickle.com>
References: 
 <CACPZYUE_NM_aUpu_JSLRtW6zjskVdvY9pF6wUau_2jnGvdvRTA@mail.gmail.com>
To: user@cassandra.apache.org

> I=92d request data, nothing would be returned, I would then re-request =
the data and it would correctly be returned:
>=20
What CL are you using for reads and writes?

> I see a number of dropped =91MUTATION=92 operations : just under 5% of =
the total =91MutationStage=92 count.
>=20
Dropped mutations in a multi DC setup may be a sign of network =
congestion or overloaded nodes.=20


> -          Could anybody suggest anything specific to look at to see =
why the repair operations aren=92t having the desired effect?=20
>=20
I would first build a test case to ensure correct operation when using =
strong consistency. i.e. QUOURM write and read. Because you are using RF =
2 per DC I assume you are not using LOCAL_QUOURM because that is 2 and =
you would not have any redundancy in the DC.=20

=20
>=20
> -          Would increasing logging level to =91DEBUG=92 show =
read-repair activity (to confirm that this is happening, when & for what =
proportion of total requests)?
It would, but the INFO logging for the AES is pretty good. I would hold =
off for now.=20

>=20
> -          Is there something obvious that I could be missing here?
When a new AES session starts it logs this

            logger.info(String.format("[repair #%s] new session: will =
sync %s on range %s for %s.%s", getName(), repairedNodes(), range, =
tablename, Arrays.toString(cfnames)));

When it completes it logs this

logger.info(String.format("[repair #%s] session completed successfully", =
getName()));

Or this on failure=20

logger.error(String.format("[repair #%s] session completed with the =
following error", getName()), exception);


Cheers

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 10/02/2013, at 9:56 PM, Brian Fleming <bigbrianfleming@gmail.com> =
wrote:

>=20
> =20
>=20
> Hi,
>=20
> =20
>=20
> I have a 20 node cluster running v1.0.7 split between 5 data centres, =
each with an RF of 2, containing a ~1TB unique dataset/~10TB of total =
data.=20
>=20
> =20
>=20
> I=92ve had some intermittent issues with a new data centre (3 nodes, =
RF=3D2) I brought online late last year with data consistency & =
availability: I=92d request data, nothing would be returned, I would =
then re-request the data and it would correctly be returned: i.e. =
read-repair appeared to be occurring.  However running repairs on the =
nodes didn=92t resolve this (I tried general =91repair=92 commands as =
well as targeted keyspace commands) =96 this didn=92t alter the =
behaviour.
>=20
> =20
>=20
> After a lot of fruitless investigation, I decided to wipe & =
re-install/re-populate the nodes.  The re-install & repair operations =
are now complete: I see the expected amount of data on the nodes, =
however I am still seeing the same behaviour, i.e. I only get data after =
one failed attempt.
>=20
> =20
>=20
> When I run repair commands, I don=92t see any errors in the logs.=20
>=20
> I see the expected =91AntiEntropySessions=92 count in =91nodetool =
tpstats=92 during repair sessions.
>=20
> I see a number of dropped =91MUTATION=92 operations : just under 5% of =
the total =91MutationStage=92 count.
>=20
> =20
>=20
> Questions :
>=20
> -          Could anybody suggest anything specific to look at to see =
why the repair operations aren=92t having the desired effect?=20
>=20
> -          Would increasing logging level to =91DEBUG=92 show =
read-repair activity (to confirm that this is happening, when & for what =
proportion of total requests)?
>=20
> -          Is there something obvious that I could be missing here?
>=20
> =20
>=20
> Many thanks,
>=20
> Brian
>=20
> =20
>=20