cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From horschi <hors...@gmail.com>
Subject Re: Stale value appears after consecutive TRUNCATE
Date Wed, 10 Aug 2016 15:41:25 GMT
Hi Yuji,

ok, perhaps you are seeing a different issue than I do.

Have you tried with durable_writes=False? If the issue is caused by the
commitlog, then it should work if you disable durable_writes.

Cheers,
Christian



On Tue, Aug 9, 2016 at 3:04 PM, Yuji Ito <yuji@imagine-orb.com> wrote:

> Thanks Christian
>
> can you reproduce the behaviour with a single node?
>
> I tried my test with a single node. But I can't.
>
> This behaviour is seems to be CQL only, or at least has gotten worse with
>> CQL. I did not experience this with Thrift.
>
> I truncate tables with CQL. I've never tried with Thrift.
>
> I think that my problem can happen when truncating even succeeds.
> That's because I check all records after truncating.
>
> I checked the source code.
> ReplayPosition.segment and position become -1 and 0 (ReplayPosition.NONE)
> in dscardSSTables() at truncating a table when there is no SSTable.
> I guess that ReplayPosition.segment shouldn't be -1 at truncating a table
> in this case.
> replayMutation() can request unexpected replay mutations because of this
> segment's value.
>
> Is there anyone familiar with truncate and replay?
>
> Regards,
> Yuji
>
>
> On Mon, Aug 8, 2016 at 6:36 PM, horschi <horschi@gmail.com> wrote:
>
>> Hi Yuji,
>>
>> can you reproduce the behaviour with a single node?
>>
>> The reason I ask is because I probably have the same issue with my
>> automated tests (which run truncate between every test), which run on my
>> local laptop.
>>
>> Maybe around 5 tests randomly fail out of my 1800. I can see that the
>> failed tests sometimes show data from other tests, which I think must be
>> because of a failed truncate. This behaviour is seems to be CQL only, or at
>> least has gotten worse with CQL. I did not experience this with Thrift.
>>
>> regards,
>> Christian
>>
>>
>>
>> On Mon, Aug 8, 2016 at 7:34 AM, Yuji Ito <yuji@imagine-orb.com> wrote:
>>
>>> Hi all,
>>>
>>> I have a question about clearing table and commit log replay.
>>> After some tables were truncated consecutively, I got some stale values.
>>> This problem doesn't occur when I clear keyspaces with DROP (and CREATE).
>>>
>>> I'm testing the following test with node failure.
>>> Some stale values appear at checking phase.
>>>
>>> Test iteration:
>>> 1. initialize tables as below
>>> 2. request a lot of read/write concurrently
>>> 3. check all records
>>> 4. repeat from the beginning
>>>
>>> I use C* 2.2.6. There are 3 nodes (replication_factor: 3).
>>> Each node kills cassandra process at random intervals and restarts it
>>> immediately.
>>>
>>> My initialization:
>>> 1. clear tables with TRUNCATE
>>> 2. INSERT initial records
>>> 3. check if all values are correct
>>>
>>> If any phase fails (because of node failure), the initialization starts
>>> all over again.
>>> So, tables are sometimes truncated consecutively.
>>> Though the check in the initialization is OK, stale data appears when I
>>> execute "SELECT * FROM mykeyspace.mytable;" after a lot of requests are
>>> completed.
>>>
>>> The problem is likely to occur when the ReplayPosition's value in
>>> "truncated_at" is initialized as below after an empty table is truncated.
>>>
>>> Column Family ID: truncated_at
>>> XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX: 0xffffffffffffffff000000000000
>>> 0156597cd4c7
>>> (this value was acquired just after phase 1 in my initialization)
>>>
>>> I guess some unexpected replays occur.
>>> Does anyone know the behavior?
>>>
>>> Thanks,
>>> Yuji
>>>
>>
>>
>

Mime
View raw message