Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@cassandra.apache.org
Date: Wed, 16 Oct 2013 15:17:42 +0000 (UTC)
From: "Constance Eustace (JIRA)" <jira@apache.org>
To: commits@cassandra.apache.org
Message-ID: <JIRA.12672030.1380749488431.73819.1381936662800@arcas>
In-Reply-To: <JIRA.12672030.1380749488431@arcas>
References: <JIRA.12672030.1380749488431@arcas>
Subject: [jira] [Commented] (CASSANDRA-6137) CQL3 SELECT IN CLAUSE
 inconsistent
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/CASSANDRA-6137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13796875#comment-13796875 ] 

Constance Eustace commented on CASSANDRA-6137:
----------------------------------------------

Here is the confirmed/known information:

Reproduction has occured in three environments. (Cassandra 2.0.1 for our Ubuntu AWS QA and Dev environments, 1.2.8 Cassandra on my local OSX laptop)

It seems to occur with large ingestions/insertion jobs combined with some update calls coming to existing data. The insertion jobs can be in totally different keyspaces from what we've seen in two of the occurrences.

The total volume here is only around a million rowkeys or so. There seems to be a bare minimum of something in the range of a million rows before we start seeing problems. Our INSERTs are done as fast as we can, usually at CL.ONE, the updates are more intermittent as processing threads detect and begin processing "Jobs".

Once we've detected problematic rows we attempted: restarts, nodetool cache invalidations, and nodetool repairs all did nothing to fix the problem. Doing a COPY TO / COPY FROM temporarily fixes the problem, but then it begins to develop again.

Knowing that :'s are used in the composition of column names, we have also tried @ signs as separators in our columns, but there are located rows/queries with all @ separators exhibiting the problem. There also seems to be a large incidence of our column names ending in "count" that are problematic, but it appears there are a few detected NOMATCHes that don't involve count columns (but there may still be disruption under the hood in those queries).

---------------------------------------------------------------------------------------------------------------------------------------- 

The problem is detected by doing this:
- SELECT rowkeys. for each rowkey:
  - SELECT columnkey FROM the rowkey to get all the columnkeys of the row (this appears to work reliably, as does SELECT *,  as seen in individual TRACE logs)
  - calculate and perform 16 different random permutations of column key sets and do SELECT * FROM table WHERE rowkey = $rowkey AND columnkey IN ($columnKeySubset)
  - if any of those fail, output the failures along with the select statements and the column keys that WERE returned. 

Different "corrupted" rowkeys will return different subsets of column keys even if they have all the column keys. So if both Row 1 and Row 2 have A B C D E, and we select B,D,E, some return B,D,E, some return BE, some return BD, some return B, etc...

you can see those NOMATCH outputs above, and samples were further confirmed through CQLSH to confirm it isn't a cassandra-jdbc driver issue.

---------------------------------------------------------------------------------------------------------------------------------------- 

Above the NOMATCHes, you can see a TRACE output of execution of these two queries seems to show the problem occurring:

You can see the Thrift calls returning all the columns/composite columns for column keys for the query without the IN clause:  

SELECT * FROM wyfr_submission.entity_job where e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' ;

__CPSYS_links
__CPSYS_name
__CPSYS_type
subPropA\:filttest\:sdf
urn\:bby\:pcm\:ingest\:status
urn\:bby\:pcm\:ingest\:status:ingeststatus
urn\:bby\:pcm\:job\:sourceparty\:reference\:id
urn@bby@pcm@job@ingest@content@complete@count

But doing 

SELECT * FROM wyfr_submission.entity_job WHERE e_entid = '924d6742-31fd-11e3-97f7-001c42000009-CJOB' AND p_prop IN ('__CPSYS_name','urn:bby:pcm:ingest:status','subPropA:filttest:sdf','urn@bby@pcm@job@ingest@content@complete@count')

__CPSYS_name
urn\:bby\:pcm\:ingest\:status
subPropA\:filttest\:sdf

... it doesn't find urn@bby@pcm@job@ingest@content@complete@count

---------------------------------------------------------------------------------------------------------------------------------------- 

The count columns such as 'urn@bby@pcm@job@ingest@content@complete@count' are not counter type columns. The actual counter values are on a different table, the column keys are just property existence indicators in an Entity-Attribute-Value datamodel. 

---------------------------------------------------------------------------------------------------------------------------------------- 

Finally, we also saw this strangeness:

SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:key')   ::   returns 0 rows (expected)
SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:key', 'exists:column:A',  'exists:column:B')   ::   returns 0 rows (sparse storage model should still return the existent column keys, right?)

BUT, doing a simple shortening of the nonexistent key by a bit: 

SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:', 'exists:column:A',  'exists:column:B')   ::   returns existent 2 rows... huh?

Then we put another random character at the end of the nonexistent, such as 'J':

SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:J', 'exists:column:A',  'exists:column:B')   ::   returns 0 rows...

But put A, B, or C and 2 rows come back:

SELECT * FROM table WHERE rowkey = $key AND columnkey IN ('nonexistent:column:C', 'exists:column:A',  'exists:column:B')   ::   returns 2 rows...

---------------------------------------------------------------------------------------------------------------------------------------- 

So this appears to be:

1) hashing related
2) uses a Range Slice at the thrift level based on tracing. Hm, this may be an assumption...
3) so this seems to indicate CASS must be starting from one point and ending at another, but is picking the wrong start and end point from the IN clause that leaves out requested columns.

This would explain why random nonexistent columns would impact the returned column keys. Manipulating the nonexistent column keys makes the starting/finishing hash change, which excludes some or all columns. Now, why this works for a while then degrades... well I speculate:

1) while data hasn't been flushed to SSTables / In memory, it all works fine, but the flush changes the hashes or something like that
2) compaction messes things up. 

---------------------------------------------------------------------------------------------------------------------------------------- 

We have it locally reproduced in an environment I can toggle controlled TRACE/DEBUG logs as well as attach Eclipse debugging to set remote breakpoints and step through. I've already done that once and will attempt a couple more times today to see what the problem may be. If you have any advice for likely areas of the code that the problem may emanate from, I would love to know. I have a couple of days I can devote to investigation, isolation, and root cause analysis...


> CQL3 SELECT IN CLAUSE inconsistent
> ----------------------------------
>
>                 Key: CASSANDRA-6137
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6137
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Ubuntu AWS Cassandra 2.0.1 SINGLE NODE on EBS RAID storage
> OSX Cassandra 1.2.8 on SSD storage
>            Reporter: Constance Eustace
>            Priority: Minor
>
> I am elevating this to Critical after doing some trace and reproducing in several environments. No one has commented on this bug from the cassandra team, and I view unreliable/corrupted data a pretty big deal. We are considering pulling cassandra and using something else.
> We have the data state reproduced locally in an environment that we can set TRACE logging, attach a debugger, etc. Some guidance as to where to look would be greatly appreciated.
> ------------------------------------------------------------------
> We are encountering inconsistent results from CQL3 queries with column keys using IN clause in WHERE. This has been reproduced in cqlsh and the jdbc driver.
> Rowkey is e_entid
> Column key is p_prop
> This returns roughly 21 rows for 21 column keys that match p_prop.
> cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB';
> These three queries each return one row for the requested single column key in the IN clause:
> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB'  AND p_prop in ('urn:bby:pcm:job:ingest:content:complete:count');
> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB'  AND p_prop in ('urn:bby:pcm:job:ingest:content:all:count');
> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB'  AND p_prop in ('urn:bby:pcm:job:ingest:content:fail:count');
> This query returns ONLY ONE ROW (one column key), not three as I would expect from the three-column-key IN clause:
> cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB'  AND p_prop in ('urn:bby:pcm:job:ingest:content:complete:count','urn:bby:pcm:job:ingest:content:all:count','urn:bby:pcm:job:ingest:content:fail:count');
> This query does return two rows however for the requested two column keys:
> cqlsh> SELECT e_entid,e_entname,e_enttype,p_prop,p_flags,p_propid,e_entlinks,p_proplinks,p_subents,p_val,p_vallinks,p_vars FROM internal_submission.Entity_Job WHERE e_entid = '845b38f1-2b91-11e3-854d-126aad0075d4-CJOB'  AND p_prop in (                                                'urn:bby:pcm:job:ingest:content:all:count','urn:bby:pcm:job:ingest:content:fail:count');
> cqlsh> describe table internal_submission.entity_job;
> CREATE TABLE entity_job (
>   e_entid text,
>   p_prop text,
>   describes text,
>   dndcondition text,
>   e_entlinks text,
>   e_entname text,
>   e_enttype text,
>   ingeststatus text,
>   ingeststatusdetail text,
>   p_flags text,
>   p_propid text,
>   p_proplinks text,
>   p_storage text,
>   p_subents text,
>   p_val text,
>   p_vallang text,
>   p_vallinks text,
>   p_valtype text,
>   p_valunit text,
>   p_vars text,
>   partnerid text,
>   referenceid text,
>   size int,
>   sourceip text,
>   submitdate bigint,
>   submitevent text,
>   userid text,
>   version text,
>   PRIMARY KEY (e_entid, p_prop)
> ) WITH
>   bloom_filter_fp_chance=0.010000 AND
>   caching='KEYS_ONLY' AND
>   comment='' AND
>   dclocal_read_repair_chance=0.000000 AND
>   gc_grace_seconds=864000 AND
>   index_interval=128 AND
>   read_repair_chance=0.100000 AND
>   replicate_on_write='true' AND
>   populate_io_cache_on_flush='false' AND
>   default_time_to_live=0 AND
>   speculative_retry='NONE' AND
>   memtable_flush_period_in_ms=0 AND
>   compaction={'class': 'SizeTieredCompactionStrategy'} AND
>   compression={'sstable_compression': 'LZ4Compressor'};
> CREATE INDEX internal_submission__JobDescribesIDX ON entity_job (describes);
> CREATE INDEX internal_submission__JobDNDConditionIDX ON entity_job (dndcondition);
> CREATE INDEX internal_submission__JobIngestStatusIDX ON entity_job (ingeststatus);
> CREATE INDEX internal_submission__JobIngestStatusDetailIDX ON entity_job (ingeststatusdetail);
> CREATE INDEX internal_submission__JobReferenceIDIDX ON entity_job (referenceid);
> CREATE INDEX internal_submission__JobUserIDX ON entity_job (userid);
> CREATE INDEX internal_submission__JobVersionIDX ON entity_job (version);
> -------------------------------
> My suspicion is that the three-column-key IN Clause is translated (improperly or not) to a two-column key range with the assumption that the third column key is present in that range, but it isn't...
> -----------------------------------
> We have tried: nodetool cache invalidations, start/stop of cassandra. Those did NOT fix the problem. A table dump (COPY TO) and then reload (COPY FROM) does fix the rows, but then more corruption creeps in.
> We are using the cassandra-jdbc driver, but I don't see anything wrong with the issued statements inside the cassandra source code when I step through the code. 
> With additional writes, it may be possible that some rows get fixed. Compaction or other jobs may repair this, but on the timescale of hours done debugging, the failures are consistent. 


--
This message was sent by Atlassian JIRA
(v6.1#6144)