Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id BF74B200D06 for ; Mon, 25 Sep 2017 11:16:48 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id BE01D1609C4; Mon, 25 Sep 2017 09:16:48 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 16D151609BB for ; Mon, 25 Sep 2017 11:16:47 +0200 (CEST) Received: (qmail 29304 invoked by uid 500); 25 Sep 2017 09:16:46 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 29294 invoked by uid 99); 25 Sep 2017 09:16:46 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 25 Sep 2017 09:16:46 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id A35231A3769 for ; Mon, 25 Sep 2017 09:16:45 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1 X-Spam-Level: * X-Spam-Status: No, score=1 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id fDZu1bfN8HoP for ; Mon, 25 Sep 2017 09:16:43 +0000 (UTC) Received: from yasigi.org (yasigi.org [77.72.144.66]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 18BE15F5FD for ; Mon, 25 Sep 2017 09:16:43 +0000 (UTC) Received: by yasigi.org (Postfix, from userid 33) id 992102ACD; Mon, 25 Sep 2017 11:16:36 +0200 (CEST) To: user@cassandra.apache.org Subject: [3.0.14] Thrift works fine, CQL/Native queries give nonsensical results X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Mon, 25 Sep 2017 11:16:36 +0200 From: Michiel Buddingh Message-ID: <45c7ae18872db077925239a207fcf9e6@michielbuddingh.net> X-Sender: michiel@michielbuddingh.net User-Agent: Roundcube Webmail/1.2.3 archived-at: Mon, 25 Sep 2017 09:16:48 -0000 LS, We are currently in the process of migrating data from an old Cassandra cluster to a new one. When quering data from a table that was copied using sstableloader, we find that even at consistency level ALL, results contain 200% duplicate entries, or worse, keep paginating and repeating themselves forever. To make matters more confusing, when we query the same tables using thrift, 3.0.14 responds promptly, and the data returned is correct and unique. We've initiated full repairs of the column family several time, but since Thrift does return consistent results, we doubt that the data itself is corrupt. Could anyone advise a way to debug this problem further? Should we investigate the table metadata, or the network layer? Thanks in advance, Michiel Attached below are the table description and trace log of a query that repeats itself forever. CREATE TABLE boring.medium_sized_table ( key text, column1 text, value varint, PRIMARY KEY (key, column1) ) WITH COMPACT STORAGE AND CLUSTERING ORDER BY (column1 DESC) AND bloom_filter_fp_chance = 0.1 AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'} AND comment = 'Contains the boring statistics per hour' AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', 'sstable_size_in_mb': '160'} AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND crc_check_chance = 1.0 AND dclocal_read_repair_chance = 0.0 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.1 AND speculative_retry = '99PERCENTILE'; READ message received from /10.10.0.70 on /10.10.0.66[MessagingService-Incoming-/10.10.0.70] at Wed Sep 20 11:09:26 CEST 2017 Parsing SELECT * FROM medium_sized_table WHERE key=?; on /10.10.0.70[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 Executing single-partition query on medium_sized_table on /10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 Preparing statement on /10.10.0.70[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 Acquiring sstable references on /10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 Read-repair GLOBAL on /10.10.0.70[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 Bloom filter allows skipping sstable 164014 on /10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 reading data from /10.10.0.66 on /10.10.0.70[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 Key cache hit for sstable 163275 on /10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 Key cache hit for sstable 163187 on /10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 Skipped 0/3 non-slice-intersecting sstables, included 0 due to tombstones on /10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 Merging data from memtables and 2 sstables on /10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 reading data from /10.10.0.78 on /10.10.0.70[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 READ message received from /10.10.0.70 on /10.10.0.78[MessagingService-Incoming-/10.10.0.70] at Wed Sep 20 11:09:26 CEST 2017 Sending READ message to /10.10.0.66 on /10.10.0.70[MessagingService-Outgoing-/10.10.0.66] at Wed Sep 20 11:09:26 CEST 2017 Executing single-partition query on medium_sized_table on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017 Executing single-partition query on medium_sized_table on /10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017 Acquiring sstable references on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017 Sending READ message to /10.10.0.78 on /10.10.0.70[MessagingService-Outgoing-/10.10.0.78] at Wed Sep 20 11:09:26 CEST 2017 Bloom filter allows skipping sstable 128083 on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017 Acquiring sstable references on /10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017 Bloom filter allows skipping sstable 127687 on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017 Bloom filter allows skipping sstable 125855 on /10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017 Key cache hit for sstable 126764 on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017 Bloom filter allows skipping sstable 124137 on /10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017 Key cache hit for sstable 127128 on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017 Key cache hit for sstable 125183 on /10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017 Key cache hit for sstable 127637 on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017 Bloom filter allows skipping sstable 124188 on /10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017 Skipped 0/5 non-slice-intersecting sstables, included 0 due to tombstones on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017 Skipped 0/4 non-slice-intersecting sstables, included 0 due to tombstones on /10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017 Merging data from memtables and 3 sstables on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017 Merging data from memtables and 1 sstables on /10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017 Read 5000 live and 0 tombstone cells on /10.10.0.70[SharedPool-Worker-6] at Wed Sep 20 11:09:26 CEST 2017 Read 5000 live and 0 tombstone cells on /10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 Enqueuing response to /10.10.0.70 on /10.10.0.66[SharedPool-Worker-8] at Wed Sep 20 11:09:26 CEST 2017 Sending REQUEST_RESPONSE message to cassh01/10.10.0.70 on /10.10.0.66[MessagingService-Outgoing-cassh01/10.10.0.70] at Wed Sep 20 11:09:26 CEST 2017 REQUEST_RESPONSE message received from /10.10.0.66 on /10.10.0.70[MessagingService-Incoming-/10.10.0.66] at Wed Sep 20 11:09:26 CEST 2017 Processing response from /10.10.0.66 on /10.10.0.70[SharedPool-Worker-7] at Wed Sep 20 11:09:26 CEST 2017 Read 5000 live and 0 tombstone cells on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017 Enqueuing response to /10.10.0.70 on /10.10.0.78[SharedPool-Worker-1] at Wed Sep 20 11:09:26 CEST 2017 Sending REQUEST_RESPONSE message to /10.10.0.70 on /10.10.0.78[MessagingService-Outgoing-/10.10.0.70] at Wed Sep 20 11:09:26 CEST 2017 REQUEST_RESPONSE message received from /10.10.0.78 on /10.10.0.70[MessagingService-Incoming-/10.10.0.78] at Wed Sep 20 11:09:26 CEST 2017 Processing response from /10.10.0.78 on /10.10.0.70[SharedPool-Worker-10] at Wed Sep 20 11:09:26 CEST 2017 Initiating read-repair on /10.10.0.70[SharedPool-Worker-10] at Wed Sep 20 11:09:26 CEST 2017 --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org For additional commands, e-mail: user-help@cassandra.apache.org