Return-Path: X-Original-To: apmail-cassandra-commits-archive@www.apache.org Delivered-To: apmail-cassandra-commits-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 96C8110BD4 for ; Sun, 8 Sep 2013 00:41:52 +0000 (UTC) Received: (qmail 13314 invoked by uid 500); 8 Sep 2013 00:41:52 -0000 Delivered-To: apmail-cassandra-commits-archive@cassandra.apache.org Received: (qmail 13295 invoked by uid 500); 8 Sep 2013 00:41:52 -0000 Mailing-List: contact commits-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@cassandra.apache.org Delivered-To: mailing list commits@cassandra.apache.org Received: (qmail 13286 invoked by uid 99); 8 Sep 2013 00:41:52 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 08 Sep 2013 00:41:52 +0000 Date: Sun, 8 Sep 2013 00:41:52 +0000 (UTC) From: "Alex Liu (JIRA)" To: commits@cassandra.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CASSANDRA-5975) Filtering on Secondary Index Takes a Long Time Even with Limit 1, Trace Log Filled with Looping Messages MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CASSANDRA-5975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13761154#comment-13761154 ] Alex Liu commented on CASSANDRA-5975: ------------------------------------- It's easy to reproduce it using a smaller set of data. > Filtering on Secondary Index Takes a Long Time Even with Limit 1, Trace Log Filled with Looping Messages > -------------------------------------------------------------------------------------------------------- > > Key: CASSANDRA-5975 > URL: https://issues.apache.org/jira/browse/CASSANDRA-5975 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Ubuntu, Single Node > Reporter: Russell Alexander Spitzer > Assignee: Aleksey Yeschenko > Fix For: 1.2.10 > > Attachments: 5975-1.2-branch.txt, 5975-v2.txt, 5975-v2.txt, create_data.py, patch.txt, trace.log > > > After creating a table with 300,000 keys. Attempting to filter on a column with a secondary index causes an rpc timeout. Using a limit statement does not alleviate the problem. The tracing log appears to be filled with the same set of messages repeated over and over until the query times out. > The data was created with the attached script and the command > {code} > python create_data.py --num-keys 300000 --num-columns 50 --keyspace 'ks' --columnfamily cf_300000_keys_50_cols --create-index y -v 3 > {code} > The query causing the delay is > {code} > select * from cf_300000_keys_50_cols where color = 'green' limit 1; > {code} > An excerpt of the trace log > {code} > Tracing session: cedbead0-14d7-11e3-915e-999f6c86239a > activity | timestamp | source | source_elapsed > -----------------------------------------------------------------------------------+--------------+--------------+---------------- > execute_cql3_query | 20:31:27,230 | 10.196.1.106 | 0 > Parsing select * from cf_300000_keys_50_cols where color = 'green' limit 1; | 20:31:27,230 | 10.196.1.106 | 31 > Peparing statement | 20:31:27,230 | 10.196.1.106 | 219 > Determining replicas to query | 20:31:27,230 | 10.196.1.106 | 563 > Executing indexed scan for [min(-9223372036854775808), min(-9223372036854775808)] | 20:31:27,232 | 10.196.1.106 | 1816 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,232 | 10.196.1.106 | 2036 > Acquiring sstable references | 20:31:27,232 | 10.196.1.106 | 2201 > Merging memtable tombstones | 20:31:27,232 | 10.196.1.106 | 2345 > Key cache hit for sstable 3 | 20:31:27,232 | 10.196.1.106 | 2493 > Seeking to partition beginning in data file | 20:31:27,232 | 10.196.1.106 | 2555 > Key cache hit for sstable 1 | 20:31:27,234 | 10.196.1.106 | 3742 > Seeking to partition beginning in data file | 20:31:27,234 | 10.196.1.106 | 3806 > Merging data from memtables and 2 sstables | 20:31:27,236 | 10.196.1.106 | 5805 > Read 3 live and 0 tombstoned cells | 20:31:27,236 | 10.196.1.106 | 5977 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,236 | 10.196.1.106 | 6166 > Acquiring sstable references | 20:31:27,236 | 10.196.1.106 | 6319 > Merging memtable tombstones | 20:31:27,236 | 10.196.1.106 | 6382 > Key cache hit for sstable 3 | 20:31:27,236 | 10.196.1.106 | 6421 > Seeking to partition beginning in data file | 20:31:27,236 | 10.196.1.106 | 6423 > Bloom filter allows skipping sstable 2 | 20:31:27,237 | 10.196.1.106 | 7060 > Bloom filter allows skipping sstable 1 | 20:31:27,237 | 10.196.1.106 | 7218 > Merging data from memtables and 1 sstables | 20:31:27,237 | 10.196.1.106 | 7358 > Read 1 live and 0 tombstoned cells | 20:31:27,238 | 10.196.1.106 | 7644 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,238 | 10.196.1.106 | 7855 > Acquiring sstable references | 20:31:27,238 | 10.196.1.106 | 8008 > Merging memtable tombstones | 20:31:27,238 | 10.196.1.106 | 8072 > Bloom filter allows skipping sstable 3 | 20:31:27,238 | 10.196.1.106 | 8225 > Bloom filter allows skipping sstable 2 | 20:31:27,238 | 10.196.1.106 | 8284 > Key cache hit for sstable 1 | 20:31:27,238 | 10.196.1.106 | 8367 > Seeking to partition beginning in data file | 20:31:27,238 | 10.196.1.106 | 8468 > Merging data from memtables and 1 sstables | 20:31:27,239 | 10.196.1.106 | 8968 > Read 1 live and 0 tombstoned cells | 20:31:27,239 | 10.196.1.106 | 9234 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,239 | 10.196.1.106 | 9405 > Acquiring sstable references | 20:31:27,239 | 10.196.1.106 | 9547 > Merging memtable tombstones | 20:31:27,240 | 10.196.1.106 | 9608 > Key cache hit for sstable 3 | 20:31:27,240 | 10.196.1.106 | 9700 > Seeking to partition indexed section in data file | 20:31:27,240 | 10.196.1.106 | 9884 > Key cache hit for sstable 1 | 20:31:27,240 | 10.196.1.106 | 10005 > Seeking to partition indexed section in data file | 20:31:27,240 | 10.196.1.106 | 10175 > Merging data from memtables and 2 sstables | 20:31:27,240 | 10.196.1.106 | 10323 > Read 3 live and 0 tombstoned cells | 20:31:27,249 | 10.196.1.106 | 19358 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,249 | 10.196.1.106 | 19516 > Acquiring sstable references | 20:31:27,249 | 10.196.1.106 | 19580 > Merging memtable tombstones | 20:31:27,250 | 10.196.1.106 | 19670 > Key cache hit for sstable 3 | 20:31:27,250 | 10.196.1.106 | 19765 > Seeking to partition beginning in data file | 20:31:27,250 | 10.196.1.106 | 19884 > Bloom filter allows skipping sstable 2 | 20:31:27,250 | 10.196.1.106 | 20357 > Bloom filter allows skipping sstable 1 | 20:31:27,250 | 10.196.1.106 | 20514 > Merging data from memtables and 1 sstables | 20:31:27,250 | 10.196.1.106 | 20576 > Read 1 live and 0 tombstoned cells | 20:31:27,251 | 10.196.1.106 | 20864 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,251 | 10.196.1.106 | 21072 > Acquiring sstable references | 20:31:27,251 | 10.196.1.106 | 21137 > Merging memtable tombstones | 20:31:27,251 | 10.196.1.106 | 21315 > Key cache hit for sstable 3 | 20:31:27,251 | 10.196.1.106 | 21461 > Seeking to partition indexed section in data file | 20:31:27,252 | 10.196.1.106 | 21599 > Key cache hit for sstable 1 | 20:31:27,252 | 10.196.1.106 | 21761 > Seeking to partition indexed section in data file | 20:31:27,252 | 10.196.1.106 | 21909 > Merging data from memtables and 2 sstables | 20:31:27,252 | 10.196.1.106 | 21977 > Read 3 live and 0 tombstoned cells | 20:31:27,261 | 10.196.1.106 | 30678 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,261 | 10.196.1.106 | 31153 > Acquiring sstable references | 20:31:27,261 | 10.196.1.106 | 31156 > Merging memtable tombstones | 20:31:27,261 | 10.196.1.106 | 31221 > Key cache hit for sstable 3 | 20:31:27,261 | 10.196.1.106 | 31259 > Seeking to partition beginning in data file | 20:31:27,261 | 10.196.1.106 | 31261 > Bloom filter allows skipping sstable 2 | 20:31:27,264 | 10.196.1.106 | 33808 > Bloom filter allows skipping sstable 1 | 20:31:27,264 | 10.196.1.106 | 33875 > Merging data from memtables and 1 sstables | 20:31:27,264 | 10.196.1.106 | 33877 > Read 1 live and 0 tombstoned cells | 20:31:27,264 | 10.196.1.106 | 34313 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,264 | 10.196.1.106 | 34488 > Acquiring sstable references | 20:31:27,264 | 10.196.1.106 | 34552 > Merging memtable tombstones | 20:31:27,265 | 10.196.1.106 | 34642 > Key cache hit for sstable 3 | 20:31:27,265 | 10.196.1.106 | 34792 > Seeking to partition indexed section in data file | 20:31:27,265 | 10.196.1.106 | 34851 > Key cache hit for sstable 1 | 20:31:27,265 | 10.196.1.106 | 35007 > Seeking to partition indexed section in data file | 20:31:27,265 | 10.196.1.106 | 35066 > Merging data from memtables and 2 sstables | 20:31:27,265 | 10.196.1.106 | 35272 > Read 3 live and 0 tombstoned cells | 20:31:27,274 | 10.196.1.106 | 44333 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,274 | 10.196.1.106 | 44529 > Acquiring sstable references | 20:31:27,275 | 10.196.1.106 | 44686 > Merging memtable tombstones | 20:31:27,275 | 10.196.1.106 | 44752 > Bloom filter allows skipping sstable 3 | 20:31:27,275 | 10.196.1.106 | 44766 > Bloom filter allows skipping sstable 2 | 20:31:27,275 | 10.196.1.106 | 45021 > Key cache hit for sstable 1 | 20:31:27,275 | 10.196.1.106 | 45163 > Seeking to partition beginning in data file | 20:31:27,275 | 10.196.1.106 | 45241 > Merging data from memtables and 1 sstables | 20:31:27,276 | 10.196.1.106 | 45719 > Read 1 live and 0 tombstoned cells | 20:31:27,276 | 10.196.1.106 | 45985 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,276 | 10.196.1.106 | 46171 > Acquiring sstable references | 20:31:27,276 | 10.196.1.106 | 46235 > Merging memtable tombstones | 20:31:27,276 | 10.196.1.106 | 46325 > Key cache hit for sstable 3 | 20:31:27,276 | 10.196.1.106 | 46501 > Seeking to partition indexed section in data file | 20:31:27,276 | 10.196.1.106 | 46561 > Key cache hit for sstable 1 | 20:31:27,277 | 10.196.1.106 | 46652 > Seeking to partition indexed section in data file | 20:31:27,277 | 10.196.1.106 | 46853 > Merging data from memtables and 2 sstables | 20:31:27,277 | 10.196.1.106 | 46922 > Read 3 live and 0 tombstoned cells | 20:31:27,286 | 10.196.1.106 | 56025 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,286 | 10.196.1.106 | 56198 > Acquiring sstable references | 20:31:27,286 | 10.196.1.106 | 56264 > Merging memtable tombstones | 20:31:27,286 | 10.196.1.106 | 56352 > Bloom filter allows skipping sstable 3 | 20:31:27,286 | 10.196.1.106 | 56439 > Bloom filter allows skipping sstable 2 | 20:31:27,286 | 10.196.1.106 | 56543 > Key cache hit for sstable 1 | 20:31:27,287 | 10.196.1.106 | 56631 > Seeking to partition beginning in data file | 20:31:27,287 | 10.196.1.106 | 56634 > Merging data from memtables and 1 sstables | 20:31:27,287 | 10.196.1.106 | 57194 > Read 1 live and 0 tombstoned cells | 20:31:27,287 | 10.196.1.106 | 57494 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,288 | 10.196.1.106 | 57672 > Acquiring sstable references | 20:31:27,288 | 10.196.1.106 | 57736 > Merging memtable tombstones | 20:31:27,288 | 10.196.1.106 | 57922 > Key cache hit for sstable 3 | 20:31:27,288 | 10.196.1.106 | 58047 > Seeking to partition indexed section in data file | 20:31:27,288 | 10.196.1.106 | 58200 > Key cache hit for sstable 1 | 20:31:27,288 | 10.196.1.106 | 58351 > Seeking to partition indexed section in data file | 20:31:27,289 | 10.196.1.106 | 58783 > Merging data from memtables and 2 sstables | 20:31:27,289 | 10.196.1.106 | 58790 > Read 3 live and 0 tombstoned cells | 20:31:27,299 | 10.196.1.106 | 69455 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,300 | 10.196.1.106 | 69641 > Acquiring sstable references | 20:31:27,300 | 10.196.1.106 | 69707 > Merging memtable tombstones | 20:31:27,300 | 10.196.1.106 | 69713 > Bloom filter allows skipping sstable 3 | 20:31:27,300 | 10.196.1.106 | 69806 > Bloom filter allows skipping sstable 2 | 20:31:27,300 | 10.196.1.106 | 69904 > Key cache hit for sstable 1 | 20:31:27,300 | 10.196.1.106 | 69994 > Seeking to partition beginning in data file | 20:31:27,300 | 10.196.1.106 | 70297 > Merging data from memtables and 1 sstables | 20:31:27,301 | 10.196.1.106 | 70778 > Read 1 live and 0 tombstoned cells | 20:31:27,301 | 10.196.1.106 | 71044 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,301 | 10.196.1.106 | 71271 > Acquiring sstable references | 20:31:27,301 | 10.196.1.106 | 71335 > Merging memtable tombstones | 20:31:27,301 | 10.196.1.106 | 71424 > Key cache hit for sstable 3 | 20:31:27,301 | 10.196.1.106 | 71583 > Seeking to partition indexed section in data file | 20:31:27,302 | 10.196.1.106 | 71645 > Key cache hit for sstable 1 | 20:31:27,302 | 10.196.1.106 | 71882 > Seeking to partition indexed section in data file | 20:31:27,302 | 10.196.1.106 | 71940 > Merging data from memtables and 2 sstables | 20:31:27,302 | 10.196.1.106 | 72030 > Read 3 live and 0 tombstoned cells | 20:31:27,311 | 10.196.1.106 | 81217 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,311 | 10.196.1.106 | 81369 > Acquiring sstable references | 20:31:27,311 | 10.196.1.106 | 81515 > Merging memtable tombstones | 20:31:27,312 | 10.196.1.106 | 81661 > Key cache hit for sstable 3 | 20:31:27,312 | 10.196.1.106 | 81812 > Seeking to partition beginning in data file | 20:31:27,312 | 10.196.1.106 | 81873 > Bloom filter allows skipping sstable 2 | 20:31:27,312 | 10.196.1.106 | 82409 > Bloom filter allows skipping sstable 1 | 20:31:27,312 | 10.196.1.106 | 82479 > Merging data from memtables and 1 sstables | 20:31:27,312 | 10.196.1.106 | 82481 > Read 1 live and 0 tombstoned cells | 20:31:27,313 | 10.196.1.106 | 82860 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,313 | 10.196.1.106 | 83035 > Acquiring sstable references | 20:31:27,313 | 10.196.1.106 | 83099 > Merging memtable tombstones | 20:31:27,313 | 10.196.1.106 | 83217 > Key cache hit for sstable 3 | 20:31:27,313 | 10.196.1.106 | 83307 > Seeking to partition indexed section in data file | 20:31:27,313 | 10.196.1.106 | 83410 > Key cache hit for sstable 1 | 20:31:27,314 | 10.196.1.106 | 83588 > Seeking to partition indexed section in data file | 20:31:27,314 | 10.196.1.106 | 83652 > Merging data from memtables and 2 sstables | 20:31:27,314 | 10.196.1.106 | 83742 > Read 3 live and 0 tombstoned cells | 20:31:27,338 | 10.196.1.106 | 108372 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,345 | 10.196.1.106 | 115180 > Acquiring sstable references | 20:31:27,345 | 10.196.1.106 | 115183 > Merging memtable tombstones | 20:31:27,345 | 10.196.1.106 | 115188 > Bloom filter allows skipping sstable 3 | 20:31:27,345 | 10.196.1.106 | 115194 > Bloom filter allows skipping sstable 2 | 20:31:27,345 | 10.196.1.106 | 115197 > Key cache hit for sstable 1 | 20:31:27,345 | 10.196.1.106 | 115203 > Seeking to partition beginning in data file | 20:31:27,345 | 10.196.1.106 | 115205 > Merging data from memtables and 1 sstables | 20:31:27,347 | 10.196.1.106 | 116821 > Read 1 live and 0 tombstoned cells | 20:31:27,348 | 10.196.1.106 | 117631 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,348 | 10.196.1.106 | 117997 > Acquiring sstable references | 20:31:27,348 | 10.196.1.106 | 118000 > Merging memtable tombstones | 20:31:27,348 | 10.196.1.106 | 118007 > Key cache hit for sstable 3 | 20:31:27,348 | 10.196.1.106 | 118014 > Seeking to partition indexed section in data file | 20:31:27,348 | 10.196.1.106 | 118017 > Key cache hit for sstable 1 | 20:31:27,348 | 10.196.1.106 | 118023 > Seeking to partition indexed section in data file | 20:31:27,348 | 10.196.1.106 | 118025 > Merging data from memtables and 2 sstables | 20:31:27,348 | 10.196.1.106 | 118030 > Read 3 live and 0 tombstoned cells | 20:31:27,356 | 10.196.1.106 | 126375 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,356 | 10.196.1.106 | 126418 > Acquiring sstable references | 20:31:27,356 | 10.196.1.106 | 126421 > Merging memtable tombstones | 20:31:27,356 | 10.196.1.106 | 126427 > Bloom filter allows skipping sstable 3 | 20:31:27,357 | 10.196.1.106 | 126695 > Bloom filter allows skipping sstable 2 | 20:31:27,357 | 10.196.1.106 | 126699 > Key cache hit for sstable 1 | 20:31:27,357 | 10.196.1.106 | 126705 > Seeking to partition beginning in data file | 20:31:27,357 | 10.196.1.106 | 126708 > Merging data from memtables and 1 sstables | 20:31:27,357 | 10.196.1.106 | 127265 > Read 1 live and 0 tombstoned cells | 20:31:27,357 | 10.196.1.106 | 127519 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,358 | 10.196.1.106 | 127686 > Acquiring sstable references | 20:31:27,358 | 10.196.1.106 | 127694 > Merging memtable tombstones | 20:31:27,358 | 10.196.1.106 | 127700 > Key cache hit for sstable 3 | 20:31:27,358 | 10.196.1.106 | 127708 > Seeking to partition indexed section in data file | 20:31:27,358 | 10.196.1.106 | 127710 > Key cache hit for sstable 1 | 20:31:27,358 | 10.196.1.106 | 127716 > Seeking to partition indexed section in data file | 20:31:27,358 | 10.196.1.106 | 127718 > Merging data from memtables and 2 sstables | 20:31:27,358 | 10.196.1.106 | 127722 > Read 3 live and 0 tombstoned cells | 20:31:27,366 | 10.196.1.106 | 135976 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,366 | 10.196.1.106 | 136018 > Acquiring sstable references | 20:31:27,366 | 10.196.1.106 | 136020 > Merging memtable tombstones | 20:31:27,366 | 10.196.1.106 | 136026 > Bloom filter allows skipping sstable 3 | 20:31:27,366 | 10.196.1.106 | 136032 > Bloom filter allows skipping sstable 2 | 20:31:27,366 | 10.196.1.106 | 136035 > Key cache hit for sstable 1 | 20:31:27,366 | 10.196.1.106 | 136040 > Seeking to partition beginning in data file | 20:31:27,366 | 10.196.1.106 | 136043 > Merging data from memtables and 1 sstables | 20:31:27,367 | 10.196.1.106 | 136852 > Read 1 live and 0 tombstoned cells | 20:31:27,367 | 10.196.1.106 | 137046 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,367 | 10.196.1.106 | 137154 > Acquiring sstable references | 20:31:27,367 | 10.196.1.106 | 137159 > Merging memtable tombstones | 20:31:27,367 | 10.196.1.106 | 137165 > Key cache hit for sstable 3 | 20:31:27,367 | 10.196.1.106 | 137172 > Seeking to partition indexed section in data file | 20:31:27,367 | 10.196.1.106 | 137175 > Key cache hit for sstable 1 | 20:31:27,367 | 10.196.1.106 | 137181 > Seeking to partition indexed section in data file | 20:31:27,367 | 10.196.1.106 | 137183 > Merging data from memtables and 2 sstables | 20:31:27,367 | 10.196.1.106 | 137187 > Read 3 live and 0 tombstoned cells | 20:31:27,375 | 10.196.1.106 | 145161 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,375 | 10.196.1.106 | 145195 > Acquiring sstable references | 20:31:27,375 | 10.196.1.106 | 145198 > Merging memtable tombstones | 20:31:27,375 | 10.196.1.106 | 145203 > Key cache hit for sstable 3 | 20:31:27,375 | 10.196.1.106 | 145212 > Seeking to partition beginning in data file | 20:31:27,375 | 10.196.1.106 | 145214 > Bloom filter allows skipping sstable 2 | 20:31:27,376 | 10.196.1.106 | 146059 > Bloom filter allows skipping sstable 1 | 20:31:27,376 | 10.196.1.106 | 146063 > Merging data from memtables and 1 sstables | 20:31:27,376 | 10.196.1.106 | 146066 > Read 1 live and 0 tombstoned cells | 20:31:27,376 | 10.196.1.106 | 146295 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,376 | 10.196.1.106 | 146416 > Acquiring sstable references | 20:31:27,376 | 10.196.1.106 | 146419 > Merging memtable tombstones | 20:31:27,376 | 10.196.1.106 | 146425 > Key cache hit for sstable 3 | 20:31:27,376 | 10.196.1.106 | 146433 > Seeking to partition indexed section in data file | 20:31:27,376 | 10.196.1.106 | 146435 > Key cache hit for sstable 1 | 20:31:27,376 | 10.196.1.106 | 146441 > Seeking to partition indexed section in data file | 20:31:27,376 | 10.196.1.106 | 146443 > Merging data from memtables and 2 sstables | 20:31:27,376 | 10.196.1.106 | 146447 > Read 3 live and 0 tombstoned cells | 20:31:27,384 | 10.196.1.106 | 153664 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,384 | 10.196.1.106 | 153708 > Acquiring sstable references | 20:31:27,384 | 10.196.1.106 | 153711 > Merging memtable tombstones | 20:31:27,384 | 10.196.1.106 | 153717 > Key cache hit for sstable 3 | 20:31:27,384 | 10.196.1.106 | 153899 > Seeking to partition beginning in data file | 20:31:27,384 | 10.196.1.106 | 153902 > Bloom filter allows skipping sstable 2 | 20:31:27,384 | 10.196.1.106 | 154556 > Bloom filter allows skipping sstable 1 | 20:31:27,384 | 10.196.1.106 | 154566 > Merging data from memtables and 1 sstables | 20:31:27,384 | 10.196.1.106 | 154568 > Read 1 live and 0 tombstoned cells | 20:31:27,385 | 10.196.1.106 | 155041 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,385 | 10.196.1.106 | 155112 > Acquiring sstable references | 20:31:27,385 | 10.196.1.106 | 155156 > Merging memtable tombstones | 20:31:27,385 | 10.196.1.106 | 155163 > Key cache hit for sstable 3 | 20:31:27,385 | 10.196.1.106 | 155171 > Seeking to partition indexed section in data file | 20:31:27,385 | 10.196.1.106 | 155173 > Key cache hit for sstable 1 | 20:31:27,385 | 10.196.1.106 | 155179 > Seeking to partition indexed section in data file | 20:31:27,385 | 10.196.1.106 | 155181 > Merging data from memtables and 2 sstables | 20:31:27,385 | 10.196.1.106 | 155185 > Read 3 live and 0 tombstoned cells | 20:31:27,393 | 10.196.1.106 | 163321 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,393 | 10.196.1.106 | 163365 > Acquiring sstable references | 20:31:27,393 | 10.196.1.106 | 163367 > Merging memtable tombstones | 20:31:27,393 | 10.196.1.106 | 163373 > Bloom filter allows skipping sstable 3 | 20:31:27,393 | 10.196.1.106 | 163379 > Bloom filter allows skipping sstable 2 | 20:31:27,393 | 10.196.1.106 | 163382 > Key cache hit for sstable 1 | 20:31:27,393 | 10.196.1.106 | 163387 > Seeking to partition beginning in data file | 20:31:27,393 | 10.196.1.106 | 163389 > Merging data from memtables and 1 sstables | 20:31:27,394 | 10.196.1.106 | 164206 > Read 1 live and 0 tombstoned cells | 20:31:27,394 | 10.196.1.106 | 164397 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,394 | 10.196.1.106 | 164562 > Acquiring sstable references | 20:31:27,394 | 10.196.1.106 | 164570 > Merging memtable tombstones | 20:31:27,394 | 10.196.1.106 | 164576 > Key cache hit for sstable 3 | 20:31:27,394 | 10.196.1.106 | 164584 > Seeking to partition indexed section in data file | 20:31:27,394 | 10.196.1.106 | 164587 > Key cache hit for sstable 1 | 20:31:27,395 | 10.196.1.106 | 164593 > Seeking to partition indexed section in data file | 20:31:27,395 | 10.196.1.106 | 164595 > Merging data from memtables and 2 sstables | 20:31:27,395 | 10.196.1.106 | 164599 > Read 3 live and 0 tombstoned cells | 20:31:27,403 | 10.196.1.106 | 172818 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,403 | 10.196.1.106 | 172860 > Acquiring sstable references | 20:31:27,403 | 10.196.1.106 | 172862 > Merging memtable tombstones | 20:31:27,403 | 10.196.1.106 | 172868 > Bloom filter allows skipping sstable 3 | 20:31:27,403 | 10.196.1.106 | 172967 > Bloom filter allows skipping sstable 2 | 20:31:27,403 | 10.196.1.106 | 172971 > Key cache hit for sstable 1 | 20:31:27,403 | 10.196.1.106 | 172979 > Seeking to partition beginning in data file | 20:31:27,403 | 10.196.1.106 | 172982 > Merging data from memtables and 1 sstables | 20:31:27,404 | 10.196.1.106 | 173735 > Read 1 live and 0 tombstoned cells | 20:31:27,404 | 10.196.1.106 | 173923 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,404 | 10.196.1.106 | 174088 > Acquiring sstable references | 20:31:27,404 | 10.196.1.106 | 174096 > Merging memtable tombstones | 20:31:27,404 | 10.196.1.106 | 174102 > Key cache hit for sstable 3 | 20:31:27,404 | 10.196.1.106 | 174110 > Seeking to partition indexed section in data file | 20:31:27,404 | 10.196.1.106 | 174112 > Key cache hit for sstable 1 | 20:31:27,404 | 10.196.1.106 | 174119 > Seeking to partition indexed section in data file | 20:31:27,404 | 10.196.1.106 | 174121 > Merging data from memtables and 2 sstables | 20:31:27,404 | 10.196.1.106 | 174124 > Read 3 live and 0 tombstoned cells | 20:31:27,414 | 10.196.1.106 | 183917 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,414 | 10.196.1.106 | 183959 > Acquiring sstable references | 20:31:27,414 | 10.196.1.106 | 183961 > Merging memtable tombstones | 20:31:27,414 | 10.196.1.106 | 183967 > Bloom filter allows skipping sstable 3 | 20:31:27,414 | 10.196.1.106 | 183973 > Bloom filter allows skipping sstable 2 | 20:31:27,414 | 10.196.1.106 | 183976 > Key cache hit for sstable 1 | 20:31:27,414 | 10.196.1.106 | 183982 > Seeking to partition beginning in data file | 20:31:27,414 | 10.196.1.106 | 183984 > Merging data from memtables and 1 sstables | 20:31:27,415 | 10.196.1.106 | 184807 > Read 1 live and 0 tombstoned cells | 20:31:27,415 | 10.196.1.106 | 184994 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,415 | 10.196.1.106 | 185105 > Acquiring sstable references | 20:31:27,415 | 10.196.1.106 | 185108 > Merging memtable tombstones | 20:31:27,415 | 10.196.1.106 | 185114 > Key cache hit for sstable 3 | 20:31:27,415 | 10.196.1.106 | 185134 > Seeking to partition indexed section in data file | 20:31:27,415 | 10.196.1.106 | 185136 > Key cache hit for sstable 1 | 20:31:27,415 | 10.196.1.106 | 185142 > Seeking to partition indexed section in data file | 20:31:27,415 | 10.196.1.106 | 185144 > Merging data from memtables and 2 sstables | 20:31:27,415 | 10.196.1.106 | 185148 > Read 3 live and 0 tombstoned cells | 20:31:27,423 | 10.196.1.106 | 193586 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,424 | 10.196.1.106 | 193635 > Acquiring sstable references | 20:31:27,424 | 10.196.1.106 | 193638 > Merging memtable tombstones | 20:31:27,424 | 10.196.1.106 | 193643 > Bloom filter allows skipping sstable 3 | 20:31:27,424 | 10.196.1.106 | 193649 > Bloom filter allows skipping sstable 2 | 20:31:27,424 | 10.196.1.106 | 193652 > Key cache hit for sstable 1 | 20:31:27,424 | 10.196.1.106 | 193657 > Seeking to partition beginning in data file | 20:31:27,424 | 10.196.1.106 | 193660 > Merging data from memtables and 1 sstables | 20:31:27,424 | 10.196.1.106 | 194274 > Read 1 live and 0 tombstoned cells | 20:31:27,424 | 10.196.1.106 | 194483 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,425 | 10.196.1.106 | 194597 > Acquiring sstable references | 20:31:27,425 | 10.196.1.106 | 194605 > Merging memtable tombstones | 20:31:27,425 | 10.196.1.106 | 194611 > Key cache hit for sstable 3 | 20:31:27,425 | 10.196.1.106 | 194619 > Seeking to partition indexed section in data file | 20:31:27,425 | 10.196.1.106 | 194621 > Key cache hit for sstable 1 | 20:31:27,425 | 10.196.1.106 | 194627 > Seeking to partition indexed section in data file | 20:31:27,425 | 10.196.1.106 | 194629 > Merging data from memtables and 2 sstables | 20:31:27,425 | 10.196.1.106 | 194633 > Read 3 live and 0 tombstoned cells | 20:31:27,433 | 10.196.1.106 | 203042 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,433 | 10.196.1.106 | 203084 > Acquiring sstable references | 20:31:27,433 | 10.196.1.106 | 203086 > Merging memtable tombstones | 20:31:27,433 | 10.196.1.106 | 203092 > Bloom filter allows skipping sstable 3 | 20:31:27,433 | 10.196.1.106 | 203097 > Bloom filter allows skipping sstable 2 | 20:31:27,433 | 10.196.1.106 | 203100 > Key cache hit for sstable 1 | 20:31:27,433 | 10.196.1.106 | 203106 > Seeking to partition beginning in data file | 20:31:27,433 | 10.196.1.106 | 203108 > Merging data from memtables and 1 sstables | 20:31:27,434 | 10.196.1.106 | 203598 > Read 1 live and 0 tombstoned cells | 20:31:27,434 | 10.196.1.106 | 203942 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,434 | 10.196.1.106 | 204112 > Acquiring sstable references | 20:31:27,434 | 10.196.1.106 | 204120 > Merging memtable tombstones | 20:31:27,434 | 10.196.1.106 | 204126 > Key cache hit for sstable 3 | 20:31:27,434 | 10.196.1.106 | 204134 > Seeking to partition indexed section in data file | 20:31:27,434 | 10.196.1.106 | 204136 > Key cache hit for sstable 1 | 20:31:27,434 | 10.196.1.106 | 204142 > Seeking to partition indexed section in data file | 20:31:27,434 | 10.196.1.106 | 204144 > Merging data from memtables and 2 sstables | 20:31:27,434 | 10.196.1.106 | 204148 > Read 3 live and 0 tombstoned cells | 20:31:27,441 | 10.196.1.106 | 211397 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,441 | 10.196.1.106 | 211439 > Acquiring sstable references | 20:31:27,441 | 10.196.1.106 | 211441 > Merging memtable tombstones | 20:31:27,441 | 10.196.1.106 | 211447 > Bloom filter allows skipping sstable 3 | 20:31:27,442 | 10.196.1.106 | 211546 > Bloom filter allows skipping sstable 2 | 20:31:27,442 | 10.196.1.106 | 211732 > Key cache hit for sstable 1 | 20:31:27,442 | 10.196.1.106 | 211737 > Seeking to partition beginning in data file | 20:31:27,442 | 10.196.1.106 | 211739 > Merging data from memtables and 1 sstables | 20:31:27,442 | 10.196.1.106 | 212484 > Read 1 live and 0 tombstoned cells | 20:31:27,443 | 10.196.1.106 | 212809 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,443 | 10.196.1.106 | 212918 > Acquiring sstable references | 20:31:27,443 | 10.196.1.106 | 212921 > Merging memtable tombstones | 20:31:27,443 | 10.196.1.106 | 212927 > Key cache hit for sstable 3 | 20:31:27,443 | 10.196.1.106 | 212934 > Seeking to partition indexed section in data file | 20:31:27,443 | 10.196.1.106 | 212936 > Key cache hit for sstable 1 | 20:31:27,443 | 10.196.1.106 | 212943 > Seeking to partition indexed section in data file | 20:31:27,443 | 10.196.1.106 | 212945 > Merging data from memtables and 2 sstables | 20:31:27,443 | 10.196.1.106 | 212949 > Read 3 live and 0 tombstoned cells | 20:31:27,451 | 10.196.1.106 | 220911 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,451 | 10.196.1.106 | 220954 > Acquiring sstable references | 20:31:27,451 | 10.196.1.106 | 220956 > Merging memtable tombstones | 20:31:27,451 | 10.196.1.106 | 220962 > Bloom filter allows skipping sstable 3 | 20:31:27,451 | 10.196.1.106 | 220968 > Bloom filter allows skipping sstable 2 | 20:31:27,451 | 10.196.1.106 | 220971 > Key cache hit for sstable 1 | 20:31:27,451 | 10.196.1.106 | 220976 > Seeking to partition beginning in data file | 20:31:27,451 | 10.196.1.106 | 220979 > Merging data from memtables and 1 sstables | 20:31:27,452 | 10.196.1.106 | 221774 > Read 1 live and 0 tombstoned cells | 20:31:27,452 | 10.196.1.106 | 221965 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,452 | 10.196.1.106 | 222093 > Acquiring sstable references | 20:31:27,452 | 10.196.1.106 | 222101 > Merging memtable tombstones | 20:31:27,452 | 10.196.1.106 | 222108 > Key cache hit for sstable 3 | 20:31:27,452 | 10.196.1.106 | 222115 > Seeking to partition indexed section in data file | 20:31:27,452 | 10.196.1.106 | 222118 > Key cache hit for sstable 1 | 20:31:27,452 | 10.196.1.106 | 222124 > Seeking to partition indexed section in data file | 20:31:27,452 | 10.196.1.106 | 222126 > Merging data from memtables and 2 sstables | 20:31:27,452 | 10.196.1.106 | 222130 > Read 3 live and 0 tombstoned cells | 20:31:27,460 | 10.196.1.106 | 230378 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,460 | 10.196.1.106 | 230420 > Acquiring sstable references | 20:31:27,460 | 10.196.1.106 | 230422 > Merging memtable tombstones | 20:31:27,460 | 10.196.1.106 | 230428 > Bloom filter allows skipping sstable 3 | 20:31:27,461 | 10.196.1.106 | 230527 > Bloom filter allows skipping sstable 2 | 20:31:27,461 | 10.196.1.106 | 230619 > Key cache hit for sstable 1 | 20:31:27,461 | 10.196.1.106 | 230624 > Seeking to partition beginning in data file | 20:31:27,461 | 10.196.1.106 | 230627 > Merging data from memtables and 1 sstables | 20:31:27,461 | 10.196.1.106 | 231223 > Read 1 live and 0 tombstoned cells | 20:31:27,461 | 10.196.1.106 | 231419 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,461 | 10.196.1.106 | 231531 > Acquiring sstable references | 20:31:27,461 | 10.196.1.106 | 231540 > Merging memtable tombstones | 20:31:27,461 | 10.196.1.106 | 231545 > Key cache hit for sstable 3 | 20:31:27,461 | 10.196.1.106 | 231553 > Seeking to partition indexed section in data file | 20:31:27,461 | 10.196.1.106 | 231555 > Key cache hit for sstable 1 | 20:31:27,461 | 10.196.1.106 | 231562 > Seeking to partition indexed section in data file | 20:31:27,461 | 10.196.1.106 | 231564 > Merging data from memtables and 2 sstables | 20:31:27,461 | 10.196.1.106 | 231568 > Read 3 live and 0 tombstoned cells | 20:31:27,470 | 10.196.1.106 | 239976 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,470 | 10.196.1.106 | 240020 > Acquiring sstable references | 20:31:27,470 | 10.196.1.106 | 240022 > Merging memtable tombstones | 20:31:27,470 | 10.196.1.106 | 240028 > Bloom filter allows skipping sstable 3 | 20:31:27,470 | 10.196.1.106 | 240034 > Bloom filter allows skipping sstable 2 | 20:31:27,470 | 10.196.1.106 | 240037 > Key cache hit for sstable 1 | 20:31:27,470 | 10.196.1.106 | 240054 > Seeking to partition beginning in data file | 20:31:27,470 | 10.196.1.106 | 240093 > Merging data from memtables and 1 sstables | 20:31:27,470 | 10.196.1.106 | 240585 > Read 1 live and 0 tombstoned cells | 20:31:27,471 | 10.196.1.106 | 240885 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,471 | 10.196.1.106 | 240993 > Acquiring sstable references | 20:31:27,471 | 10.196.1.106 | 240997 > Merging memtable tombstones | 20:31:27,471 | 10.196.1.106 | 241003 > Key cache hit for sstable 3 | 20:31:27,471 | 10.196.1.106 | 241011 > Seeking to partition indexed section in data file | 20:31:27,471 | 10.196.1.106 | 241013 > Key cache hit for sstable 1 | 20:31:27,471 | 10.196.1.106 | 241019 > Seeking to partition indexed section in data file | 20:31:27,471 | 10.196.1.106 | 241020 > Merging data from memtables and 2 sstables | 20:31:27,471 | 10.196.1.106 | 241028 > Read 3 live and 0 tombstoned cells | 20:31:27,479 | 10.196.1.106 | 249225 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,479 | 10.196.1.106 | 249265 > Acquiring sstable references | 20:31:27,479 | 10.196.1.106 | 249267 > Merging memtable tombstones | 20:31:27,479 | 10.196.1.106 | 249273 > Bloom filter allows skipping sstable 3 | 20:31:27,479 | 10.196.1.106 | 249279 > Bloom filter allows skipping sstable 2 | 20:31:27,479 | 10.196.1.106 | 249282 > Key cache hit for sstable 1 | 20:31:27,479 | 10.196.1.106 | 249287 > Seeking to partition beginning in data file | 20:31:27,479 | 10.196.1.106 | 249289 > Merging data from memtables and 1 sstables | 20:31:27,479 | 10.196.1.106 | 249296 > Read 1 live and 0 tombstoned cells | 20:31:27,479 | 10.196.1.106 | 249558 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,480 | 10.196.1.106 | 249655 > Acquiring sstable references | 20:31:27,480 | 10.196.1.106 | 249658 > Merging memtable tombstones | 20:31:27,480 | 10.196.1.106 | 249755 > Key cache hit for sstable 3 | 20:31:27,480 | 10.196.1.106 | 249763 > Seeking to partition indexed section in data file | 20:31:27,480 | 10.196.1.106 | 249765 > Key cache hit for sstable 1 | 20:31:27,480 | 10.196.1.106 | 249772 > Seeking to partition indexed section in data file | 20:31:27,480 | 10.196.1.106 | 249774 > Merging data from memtables and 2 sstables | 20:31:27,480 | 10.196.1.106 | 249778 > Read 3 live and 0 tombstoned cells | 20:31:27,487 | 10.196.1.106 | 257542 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,487 | 10.196.1.106 | 257575 > Acquiring sstable references | 20:31:27,488 | 10.196.1.106 | 257610 > Merging memtable tombstones | 20:31:27,488 | 10.196.1.106 | 257616 > Bloom filter allows skipping sstable 3 | 20:31:27,488 | 10.196.1.106 | 257622 > Bloom filter allows skipping sstable 2 | 20:31:27,488 | 10.196.1.106 | 257625 > Key cache hit for sstable 1 | 20:31:27,488 | 10.196.1.106 | 257631 > Seeking to partition beginning in data file | 20:31:27,488 | 10.196.1.106 | 257633 > Merging data from memtables and 1 sstables | 20:31:27,488 | 10.196.1.106 | 258473 > Read 1 live and 0 tombstoned cells | 20:31:27,489 | 10.196.1.106 | 259067 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,489 | 10.196.1.106 | 259197 > Acquiring sstable references | 20:31:27,489 | 10.196.1.106 | 259200 > Merging memtable tombstones | 20:31:27,489 | 10.196.1.106 | 259206 > Key cache hit for sstable 3 | 20:31:27,489 | 10.196.1.106 | 259214 > Seeking to partition indexed section in data file | 20:31:27,489 | 10.196.1.106 | 259216 > Key cache hit for sstable 1 | 20:31:27,489 | 10.196.1.106 | 259222 > Seeking to partition indexed section in data file | 20:31:27,489 | 10.196.1.106 | 259224 > Merging data from memtables and 2 sstables | 20:31:27,489 | 10.196.1.106 | 259319 > Read 3 live and 0 tombstoned cells | 20:31:27,497 | 10.196.1.106 | 267071 > Executing single-partition query on cf_300000_keys_50_cols | 20:31:27,497 | 10.196.1.106 | 267135 > Acquiring sstable references | 20:31:27,497 | 10.196.1.106 | 267137 > Merging memtable tombstones | 20:31:27,497 | 10.196.1.106 | 267143 > Key cache hit for sstable 3 | 20:31:27,497 | 10.196.1.106 | 267151 > Seeking to partition beginning in data file | 20:31:27,497 | 10.196.1.106 | 267154 > Bloom filter allows skipping sstable 2 | 20:31:27,498 | 10.196.1.106 | 267943 > Bloom filter allows skipping sstable 1 | 20:31:27,498 | 10.196.1.106 | 267957 > Merging data from memtables and 1 sstables | 20:31:27,498 | 10.196.1.106 | 267960 > Read 1 live and 0 tombstoned cells | 20:31:27,498 | 10.196.1.106 | 268176 > Executing single-partition query on cf_300000_keys_50_cols.color | 20:31:27,498 | 10.196.1.106 | 268271 > Acquiring sstable references | 20:31:27,498 | 10.196.1.106 | 268274 > Merging memtable tombstones | 20:31:27,498 | 10.196.1.106 | 268280 > Key cache hit for sstable 3 | 20:31:27,498 | > ... > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira