cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michiel Buddingh <mich...@michielbuddingh.net>
Subject [3.0.14] Thrift works fine, CQL/Native queries give nonsensical results
Date Mon, 25 Sep 2017 09:16:36 GMT
LS,

We are currently in the process of migrating data from an old Cassandra 
cluster to a new one.  When quering data from a table that was copied 
using sstableloader, we find that even at consistency level ALL, results 
contain 200% duplicate entries, or worse, keep paginating and repeating 
themselves forever.

To make matters more confusing, when we query the same tables using 
thrift, 3.0.14 responds promptly, and the data returned is correct and 
unique.

We've initiated full repairs of the column family several time, but 
since Thrift does return consistent results, we doubt that the data 
itself is corrupt.

Could anyone advise a way to debug this problem further?  Should we 
investigate the table metadata, or the network layer?

Thanks in advance,
Michiel

Attached below are the table description and trace log of a query that 
repeats itself forever.

CREATE TABLE boring.medium_sized_table (
     key text,
     column1 text,
     value varint,
     PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE
     AND CLUSTERING ORDER BY (column1 DESC)
     AND bloom_filter_fp_chance = 0.1
     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
     AND comment = 'Contains the boring statistics per hour'
     AND compaction = {'class': 
'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', 
'sstable_size_in_mb': '160'}
     AND compression = {'chunk_length_in_kb': '64', 'class': 
'org.apache.cassandra.io.compress.LZ4Compressor'}
     AND crc_check_chance = 1.0
     AND dclocal_read_repair_chance = 0.0
     AND default_time_to_live = 0
     AND gc_grace_seconds = 864000
     AND max_index_interval = 2048
     AND memtable_flush_period_in_ms = 0
     AND min_index_interval = 128
     AND read_repair_chance = 0.1
     AND speculative_retry = '99PERCENTILE';


READ message received from /10.10.0.70 on 
/10.10.0.66[MessagingService-Incoming-/10.10.0.70] at Wed Sep 20 
11:09:26 CEST 2017
Parsing SELECT * FROM medium_sized_table WHERE key=?; on 
/10.10.0.70[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017
Executing single-partition query on medium_sized_table on 
/10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017
Preparing statement on /10.10.0.70[SharedPool-Worker-8] at Wed Sep 20 
11:09:26 CEST 2017
Acquiring sstable references on /10.10.0.66[SharedPool-Worker-8] at Wed 
Sep 20 11:09:26 CEST 2017
Read-repair GLOBAL on /10.10.0.70[SharedPool-Worker-8] at Wed Sep 20 
11:09:26 CEST 2017
Bloom filter allows skipping sstable 164014 on 
/10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017
reading data from /10.10.0.66 on /10.10.0.70[SharedPool-Worker-8] at Wed 
Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 163275 on /10.10.0.66[SharedPool-Worker-8] at 
Wed Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 163187 on /10.10.0.66[SharedPool-Worker-8] at 
Wed Sep 20 11:09:26 CEST 2017
Skipped 0/3 non-slice-intersecting sstables, included 0 due to 
tombstones on /10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 
CEST 2017
Merging data from memtables and 2 sstables on 
/10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017
reading data from /10.10.0.78 on /10.10.0.70[SharedPool-Worker-8] at Wed 
Sep 20 11:09:26 CEST 2017
READ message received from /10.10.0.70 on 
/10.10.0.78[MessagingService-Incoming-/10.10.0.70] at Wed Sep 20 
11:09:26 CEST 2017
Sending READ message to /10.10.0.66 on 
/10.10.0.70[MessagingService-Outgoing-/10.10.0.66] at Wed Sep 20 
11:09:26 CEST 2017
Executing single-partition query on medium_sized_table on 
/10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017
Executing single-partition query on medium_sized_table on 
/10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017
Acquiring sstable references on /10.10.0.78[SharedPool-Worker-1] at Wed 
Sep 20 11:09:26 CEST 2017
Sending READ message to /10.10.0.78 on 
/10.10.0.70[MessagingService-Outgoing-/10.10.0.78] at Wed Sep 20 
11:09:26 CEST 2017
Bloom filter allows skipping sstable 128083 on 
/10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017
Acquiring sstable references on /10.10.0.70[SharedPool-Worker-6] at Wed 
Sep 20 11:09:26 CEST 2017
Bloom filter allows skipping sstable 127687 on 
/10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017
Bloom filter allows skipping sstable 125855 on 
/10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 126764 on /10.10.0.78[SharedPool-Worker-1] at 
Wed Sep 20 11:09:26 CEST 2017
Bloom filter allows skipping sstable 124137 on 
/10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 127128 on /10.10.0.78[SharedPool-Worker-1] at 
Wed Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 125183 on /10.10.0.70[SharedPool-Worker-6] at 
Wed Sep 20 11:09:26 CEST 2017
Key cache hit for sstable 127637 on /10.10.0.78[SharedPool-Worker-1] at 
Wed Sep 20 11:09:26 CEST 2017
Bloom filter allows skipping sstable 124188 on 
/10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017
Skipped 0/5 non-slice-intersecting sstables, included 0 due to 
tombstones on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 
CEST 2017
Skipped 0/4 non-slice-intersecting sstables, included 0 due to 
tombstones on /10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 
CEST 2017
Merging data from memtables and 3 sstables on 
/10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017
Merging data from memtables and 1 sstables on 
/10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017
Read 5000 live and 0 tombstone cells on /10.10.0.70[SharedPool-Worker-6] 
at Wed Sep 20 11:09:26 CEST 2017
Read 5000 live and 0 tombstone cells on /10.10.0.66[SharedPool-Worker-8] 
at Wed Sep 20 11:09:26 CEST 2017
Enqueuing response to /10.10.0.70 on /10.10.0.66[SharedPool-Worker-8] at 
Wed Sep 20 11:09:26 CEST 2017
Sending REQUEST_RESPONSE message to cassh01/10.10.0.70 on 
/10.10.0.66[MessagingService-Outgoing-cassh01/10.10.0.70] at Wed Sep 20 
11:09:26 CEST 2017
REQUEST_RESPONSE message received from /10.10.0.66 on 
/10.10.0.70[MessagingService-Incoming-/10.10.0.66] at Wed Sep 20 
11:09:26 CEST 2017
Processing response from /10.10.0.66 on /10.10.0.70[SharedPool-Worker-7] 
at Wed Sep 20 11:09:26 CEST 2017
Read 5000 live and 0 tombstone cells on /10.10.0.78[SharedPool-Worker-1] 
at Wed Sep 20 11:09:26 CEST 2017
Enqueuing response to /10.10.0.70 on /10.10.0.78[SharedPool-Worker-1] at 
Wed Sep 20 11:09:26 CEST 2017
Sending REQUEST_RESPONSE message to /10.10.0.70 on 
/10.10.0.78[MessagingService-Outgoing-/10.10.0.70] at Wed Sep 20 
11:09:26 CEST 2017
REQUEST_RESPONSE message received from /10.10.0.78 on 
/10.10.0.70[MessagingService-Incoming-/10.10.0.78] at Wed Sep 20 
11:09:26 CEST 2017
Processing response from /10.10.0.78 on 
/10.10.0.70[SharedPool-Worker-10] at Wed Sep 20 11:09:26 CEST 2017
Initiating read-repair on /10.10.0.70[SharedPool-Worker-10] at Wed Sep 
20 11:09:26 CEST 2017

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org


Mime
View raw message