incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From aaron morton <aa...@thelastpickle.com>
Subject Re: Cass returns Incorrect column data on writes during flushing
Date Thu, 31 Jan 2013 00:49:53 GMT
The looks bug like, can you create a ticket on https://issues.apache.org/jira/browse/CASSANDRA

Please include the C* version, the table and insert statements, and if you can repo is using
CQL 3. 

Thanks
Aaron

-----------------
Aaron Morton
Freelance Cassandra Developer
New Zealand

@aaronmorton
http://www.thelastpickle.com

On 30/01/2013, at 8:10 AM, Elden Bishop <ebishop@exacttarget.com> wrote:

> Sure thing, Here is a console dump showing the error. Notice that column '9801' is NOT
NULL on the first two queries but IS NULL on the last query. I get this behavior constantly
on any writes that coincide with a flush. The column is always readable by itself but disappears
depending on the other columns being queried.
> 
> $
> $ bin/cqlsh –2
> cqlsh>
> cqlsh> SELECT '9801' FROM BUGS.Test WHERE KEY='a';
> 
>  9801
> ---------------------
>  0.02271159951509616
> 
> cqlsh> SELECT '9801','6814' FROM BUGS.Test WHERE KEY='a';
> 
>  9801                | 6814
> ---------------------+--------------------
>  0.02271159951509616 | 0.6612351709326891
> 
> cqlsh> SELECT '9801','6814','3333' FROM BUGS.Test WHERE KEY='a';
> 
>  9801 | 6814               | 3333
> ------+--------------------+--------------------
>  null | 0.6612351709326891 | 0.8921380283891902
> 
> cqlsh> exit;
> $
> $
> 
> From: aaron morton <aaron@thelastpickle.com>
> Reply-To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Date: Tuesday, January 29, 2013 12:21 AM
> To: "user@cassandra.apache.org" <user@cassandra.apache.org>
> Subject: Re: Cass returns Incorrect column data on writes during flushing
> 
>> Ie. Query for a single column works but the column does not appear in slice queries
depending on the other columns in the query
>> 
>> cfq.getKey("foo").getColumn("A") returns "A"
>> cfq.getKey("foo").withColumnSlice("A", "B") returns "B" only
>> cfq.getKey("foo").withColumnSlice("A","B","C") returns "A","B" and "C"
> Can you replicate this using cassandra-cli or CQL ? 
> Makes it clearer what's happening and removes any potential issues with the client or
your code.
> If you cannot repo it show you astynax code.
>  
> Cheers
> 
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> New Zealand
> 
> @aaronmorton
> http://www.thelastpickle.com
> 
> On 29/01/2013, at 1:15 PM, Elden Bishop <ebishop@exacttarget.com> wrote:
> 
>> I'm trying to track down some really worrying behavior. It appears that writing multiple
columns while a table flush is occurring can result in Cassandra recording its data in a way
that makes columns visible only to some queries but not others.
>> 
>> Ie. Query for a single column works but the column does not appear in slice queries
depending on the other columns in the query
>> 
>> cfq.getKey("foo").getColumn("A") returns "A"
>> cfq.getKey("foo").withColumnSlice("A", "B") returns "B" only
>> cfq.getKey("foo").withColumnSlice("A","B","C") returns "A","B" and "C"
>> 
>> This is a permanent condition meaning that even hours later with no reads or writes
the DB will return the same results. I can reproduce this 100% of the time by writing multiple
columns and then reading a different set of multiple columns. Columns written during the flush
may or may not appear.
>> 
>> Details
>> 
>> # There are no log errors
>> # All single column queries return correct data.
>> # Slice queries may or may not return the column depending on which other columns
are in the query.
>> # This is on a stock "unzip and run" installation of Cassandra using default options
only; basically doing the cassandra getting started tutorial and using the Demo table described
in that tutorial.
>> # Cassandra 1.2.0 using Astynax and Java 1.6.0_37.
>> # There are no errors but there is always a "flushing high traffic column family"
that happens right before the incoherent state occurs
>> # to reproduce just update multiple columns at the same time, using random rows and
then verify the writes by reading multiple columns. I get can generate the error on 100% of
runs. Once the state is screwed up, the multi column read will not contain the column but
the single column read will.
>> 
>> Log snippet
>>  INFO 15:47:49,066 GC for ParNew: 320 ms for 1 collections, 207199992 used; max is
1052770304
>>  INFO 15:47:58,076 GC for ParNew: 330 ms for 1 collections, 232839680 used; max is
1052770304
>>  INFO 15:48:00,374 flushing high-traffic column family CFS(Keyspace='BUGS', ColumnFamily='Test')
(estimated 50416978 bytes)
>>  INFO 15:48:00,374 Enqueuing flush of Memtable-Test@1575891161(4529586/50416978 serialized/live
bytes, 279197 ops)
>>  INFO 15:48:00,378 Writing Memtable-Test@1575891161(4529586/50416978 serialized/live
bytes, 279197 ops)
>>  INFO 15:48:01,142 GC for ParNew: 654 ms for 1 collections, 239478568 used; max is
1052770304
>>  INFO 15:48:01,474 Completed flushing /var/lib/cassandra/data/BUGS/Test/BUGS-Test-ia-45-Data.db
(4580066 bytes) for commitlog position ReplayPosition(segmentId=1359415964165, position=7462737)
>> 
>> 
>> Any ideas on what could be going on? I could not find anything like this in the open
bugs and the only workaround seems to be never doing multi-column reads or writes. I'm concerned
that the DB can get into a state where different queries can return such inconsistent results.
All with no warning or errors. There is no way to even verify data correctness; every column
can seem correct when queried and then disappear during slice queries depending on the other
columns in the query.
>> 
>> 
>> Thanks
> 


Mime
View raw message