incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cassandra read reapair
Date Tue, 28 May 2013 18:37:40 GMT
Start using QUOURM for reads and writes and then run a nodetool repair. 

That should get you back to the land of the consistent. 

Cheers

-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 27/05/2013, at 10:01 PM, Kais Ahmed <kais@neteck-fr.com> wrote:

> Hi aaron,
> 
> I was sure that  phpcassa use QUORUM for W and R by default, but you're right, the default
CL for R AND W is ONE.
> 
> We are in this configuration W + R < N, how can i do to repair some keys that always
return inconsistent data.
> 
> Thanks,
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 2013/5/24 Kais Ahmed <kais@neteck-fr.com>
> Hi aaron an thanks,
> 
> 
> > If you are reading and writing at CL QUOURM and getting inconsistent results that
sounds like a bug. If you are mixing the CL levels such that R + W <= N then it's expected
behaviour. 
> I think it's a bug, it concern only some keys (~200 over 120 000 keys) on one column
family, 
> 
> As I understand it, if  R+W <= N it's expected behaviour at a moment, but read repair
will correct the inconsistent data (related to read_repair_chance value)
> and the next query will return a consistent data , right ?
> 
> Here is an exemple of a result that i have on a key (keyspace RF 3), 3 differents replicas
:
> 
> [default@prod] ASSUME contacts_timeordered KEYS AS ascii;
> 
> [default@prod] get contacts_timeordered['1425185-IGNORED'];
> => (column=1a927740-97ec-11e2-ab16-a1afd66a735e, value=363838353936, timestamp=1364505108842098)
> => (column=1a93d5b0-97ec-11e2-888c-2bf068e0f754, value=31373930303330, timestamp=1364505108851088)
> => (column=b5c559c0-9d0f-11e2-8682-f7ecd4112689, value=32343130303930, timestamp=1365070157421869)
> => (column=7ba22b90-b48b-11e2-a2c2-914573921d9f, value=32353031353039, timestamp=1367652194221857)
> => (column=63ef5d80-b7e8-11e2-abf8-593c289227cd, value=32383435323830, timestamp=1368021951146575)
> => (column=d6383fc0-b810-11e2-a880-bd2ecacbaee3, value=31363334363737, timestamp=1368039322753824)
> => (column=f47d8e60-bd3f-11e2-88f4-533a93fe9432, value=32373938313038, timestamp=1368609315699785)
> => (column=c5bfe060-bf8e-11e2-ab1f-07be407aff58, value=32333634353034, timestamp=1368863069848610)
> => (column=f07ae4b0-c42f-11e2-8064-9794e872eb2b, value=363838353936, timestamp=1369372095163129)
> Returned 9 results.
> Elapsed time: 10 msec(s).
> 
> [default@prod] get contacts_timeordered['1425185-IGNORED'];
> => (column=b5c559c0-9d0f-11e2-8682-f7ecd4112689, value=32343130303930, timestamp=1365070157421869)
> => (column=7ba22b90-b48b-11e2-a2c2-914573921d9f, value=32353031353039, timestamp=1367652194221857)
> => (column=63ef5d80-b7e8-11e2-abf8-593c289227cd, value=32383435323830, timestamp=1368021951146575)
> => (column=d6383fc0-b810-11e2-a880-bd2ecacbaee3, value=31363334363737, timestamp=1368039322753824)
> => (column=f47d8e60-bd3f-11e2-88f4-533a93fe9432, value=32373938313038, timestamp=1368609315699785)
> => (column=c5bfe060-bf8e-11e2-ab1f-07be407aff58, value=32333634353034, timestamp=1368863069848610)
> => (column=f07ae4b0-c42f-11e2-8064-9794e872eb2b, value=363838353936, timestamp=1369372095163129)
> Returned 7 results.
> Elapsed time: 7.49 msec(s).
> 
> [default@prod] get contacts_timeordered['1425185-IGNORED'];
> => (column=1a93d5b0-97ec-11e2-888c-2bf068e0f754, value=31373930303330, timestamp=1364505108851088)
> => (column=b5c559c0-9d0f-11e2-8682-f7ecd4112689, value=32343130303930, timestamp=1365070157421869)
> => (column=7ba22b90-b48b-11e2-a2c2-914573921d9f, value=32353031353039, timestamp=1367652194221857)
> => (column=63ef5d80-b7e8-11e2-abf8-593c289227cd, value=32383435323830, timestamp=1368021951146575)
> => (column=d6383fc0-b810-11e2-a880-bd2ecacbaee3, value=31363334363737, timestamp=1368039322753824)
> => (column=f47d8e60-bd3f-11e2-88f4-533a93fe9432, value=32373938313038, timestamp=1368609315699785)
> => (column=c5bfe060-bf8e-11e2-ab1f-07be407aff58, value=32333634353034, timestamp=1368863069848610)
> => (column=f07ae4b0-c42f-11e2-8064-9794e872eb2b, value=363838353936, timestamp=1369372095163129)
> Returned 8 results.
> Elapsed time: 9.37 msec(s).
> 
> Do I have to change read_repair_chance to 1 to correct the inconsistency, nodetool repair
don't solve it.
> 
> Thanks a lot,
> 
> 
> 
> 
> 
> 2013/5/23 aaron morton <aaron@thelastpickle.com>
> If you are reading and writing at CL QUOURM and getting inconsistent results that sounds
like a bug. If you are mixing the CL levels such that R + W <= N then it's expected behaviour.

> 
> 
> Can you reproduce the issue outside of your app ? 
> 
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Consultant
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 21/05/2013, at 8:55 PM, Kais Ahmed <kais@neteck-fr.com> wrote:
> 
>> > Checking you do not mean the row key is corrupt and cannot be read. 
>> Yes, i can read it but all read don't return the same result except for CL ALL
>> 
>> > By default in 1.X and beyond the default read repair chance is 0.1, so it's
only enabled on 10% of requests. 
>> You are right read repair chance is set to 0.1, but i launched a read repair which
did not solved the problem. Any idea?
>> 
>> >What CL are you writing at ? 
>> All write are in CL QUORUM
>> 
>> thank you aaron for your answer. 
>> 
>> 
>> 2013/5/21 aaron morton <aaron@thelastpickle.com>
>>> Only some keys of one CF are corrupt. 
>> Checking you do not mean the row key is corrupt and cannot be read. 
>> 
>>> I thought using CF ALL, would correct the problem with READ REPAIR, but by returning
to CL QUORUM, the problem persists.
>>> 
>> 
>> By default in 1.X and beyond the default read repair chance is 0.1, so it's only
enabled on 10% of requests. 
>> 
>> 
>> In the absence of further writes all reads (at any CL) should return the same value.

>> 
>> What CL are you writing at ? 
>> 
>> Cheers
>> 
>> -----------------
>> Aaron Morton
>> Freelance Cassandra Consultant
>> New Zealand
>> 
>> @aaronmorton
>> http://www.thelastpickle.com
>> 
>> On 19/05/2013, at 1:28 AM, Kais Ahmed <kais@neteck-fr.com> wrote:
>> 
>>> Hi all,
>>> 
>>> I encountered a consistency problem one some keys using phpcassa and Cassandra
1.2.3 since a server crash 
>>> 
>>> Only some keys of one CF are corrupt. 
>>> 
>>> I lauched a nodetool repair that successfully completed but don't correct the
issue.
>>> 
>>> 
>>> 
>>> When i try to get a corrupt Key with :
>>> 
>>> CL ONE, the result contains 7 or 8 or 9 columns
>>> 
>>> CL QUORUM, result contains 8 or 9 columns
>>> 
>>> CL ALL, the data is consistent and returns always 9 columns
>>> 
>>> 
>>> 
>>> I thought using CF ALL, would correct the problem with READ REPAIR, but by returning
to CL QUORUM, the problem persists.
>>> 
>>> 
>>> Thank you for your help
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> 


Mime
View raw message