cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "xiaost (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-11044) query under certain partition key takes much more time than expected
Date Thu, 21 Jan 2016 08:06:39 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15110251#comment-15110251
] 

xiaost edited comment on CASSANDRA-11044 at 1/21/16 8:06 AM:
-------------------------------------------------------------

{noformat}
"SharedPool-Worker-3" #371 daemon prio=5 os_prio=0 tid=0x00007fd3412ad000 nid=0x3bad runnable
[0x00007fd21f687000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:649)
        at org.apache.cassandra.db.RangeTombstoneList.add(RangeTombstoneList.java:170)
        at org.apache.cassandra.db.RangeTombstoneList.add(RangeTombstoneList.java:142)
        at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:239)
        at org.apache.cassandra.db.ArrayBackedSortedColumns.delete(ArrayBackedSortedColumns.java:490)
        at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:153)
        at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:189)
        at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:161)
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:89)
        at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:48)
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:83)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:70)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:317)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:2011)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1815)
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:360)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)
        at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:38)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

here is the stack. I investigated into the code and found that {{gatherTombstones.getNext}}
 sometimes iters 9000+ times and takes 5s in {{returnCF.addAtom(atom)}}

I am trying to set {{gc_grace_seconds}} with a small value and compact and table.


was (Author: xiaost):
{noformat}
"SharedPool-Worker-3" #371 daemon prio=5 os_prio=0 tid=0x00007fd3412ad000 nid=0x3bad runnable
[0x00007fd21f687000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.cassandra.db.RangeTombstoneList.insertFrom(RangeTombstoneList.java:649)
        at org.apache.cassandra.db.RangeTombstoneList.add(RangeTombstoneList.java:170)
        at org.apache.cassandra.db.RangeTombstoneList.add(RangeTombstoneList.java:142)
        at org.apache.cassandra.db.DeletionInfo.add(DeletionInfo.java:239)
        at org.apache.cassandra.db.ArrayBackedSortedColumns.delete(ArrayBackedSortedColumns.java:490)
        at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:153)
        at org.apache.cassandra.db.filter.QueryFilter$2.getNext(QueryFilter.java:189)
        at org.apache.cassandra.db.filter.QueryFilter$2.hasNext(QueryFilter.java:161)
        at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:146)
        at org.apache.cassandra.utils.MergeIterator$ManyToOne.<init>(MergeIterator.java:89)
        at org.apache.cassandra.utils.MergeIterator.get(MergeIterator.java:48)
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:107)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:83)
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:70)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:317)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:61)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:2011)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1815)
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:360)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:85)
        at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:38)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:164)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:136)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105)
        at java.lang.Thread.run(Thread.java:745)
{noformat}

here is the stack. I investigated into the code and found that {{gatherTombstones.getNext}}
 sometimes iters 9000+ times and takes 5s in {{returnCF.addAtom(atom)}}

I am trying to set gc_grace_seconds with a small value and compact and table.

> query under certain partition key takes much more time than expected 
> ---------------------------------------------------------------------
>
>                 Key: CASSANDRA-11044
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11044
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Local Write-Read Paths
>            Reporter: xiaost
>         Attachments: tracing.log
>
>
> In my cluster, all the nodes is under low workload, 
> but query under certain partition key (we found one) takes much more time than expected.

> we write & updates about 3 times per row in one day,
> reads are much more than writes.
> HARDWARD:
> 6*nodes(E5-2630, 1*ssd with 5GB data)
> TABLE DESCRIBE:
> {noformat}
> CREATE TABLE album.user_updates (
>     user_id bigint,
>     time_uuid bigint,
>     key ascii,
>     PRIMARY KEY (user_id, time_uuid)
> ) WITH CLUSTERING ORDER BY (time_uuid ASC)
>     AND bloom_filter_fp_chance = 0.01
>     AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
>     AND comment = ''
>     AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'}
>     AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'}
>     AND dclocal_read_repair_chance = 0.1
>     AND default_time_to_live = 0
>     AND gc_grace_seconds = 864000
>     AND max_index_interval = 2048
>     AND memtable_flush_period_in_ms = 0
>     AND min_index_interval = 128
>     AND read_repair_chance = 0.0
>     AND speculative_retry = '99.0PERCENTILE';
> {noformat}
> QUERYs:
> {noformat}
> select * from user_updates where user_id = 1432138730701829 limit 100;
> select count(1) from user_updates where user_id = 1432138730701829;
> {noformat}
> RESULT:  (takes about 3.5 minutes)
> {noformat}
>  count
> -------
>   1058
> (1 rows)
> {noformat}
> check attachments for the tracing log 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message