cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom van der Woerdt (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CASSANDRA-9728) Very bad read performance after 2.1.5->2.1.7 upgrade
Date Sat, 04 Jul 2015 16:55:04 GMT

     [ https://issues.apache.org/jira/browse/CASSANDRA-9728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Tom van der Woerdt updated CASSANDRA-9728:
------------------------------------------
    Description: 
After upgrading from 2.1.5 to 2.1.7, read performance has significantly slowed down for queries
that use indexes. A query that used to take under a second to execute now takes 8 seconds
or longer.

The table I am noticing this on is a fairly large table (possibly pushing Cassandra's limits
a bit, with around 1TB per node) with a single index.

{noformat}
CREATE TABLE the_table (
    id bigint PRIMARY KEY,
    id_2 bigint,
    field_1 ascii,
    field_2 blob, -- "Wide" column in that it's on average 15KB
    field_3 ascii,
    field_4 ascii,
    field_5 text,
    field_6 ascii
)  WITH compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
    AND compression = {'chunk_length_kb': '1024', 'sstable_compression': 'org.apache.cassandra.io.compress.DeflateCompressor'};
CREATE INDEX the_table_id_2_idx ON the_table (id_2);
{noformat}

Compaction status, according to nodetool's cfstats :

index:                SSTables in each level: [1714/4, 7, 0, 0, 0, 0, 0, 0, 0]
main:                SSTables in each level: [79/4, 1, 105/100, 1049/1000, 172, 0, 0, 0, 0]

It's had a few days to catch up, but the indexes don't seem to be catching up well. Do note
that the performance issue I'm reporting was also experienced when the main table had not
yet caught up. Queries fetching by the primary key have been equally fast before and after
the compaction catch-up.

One thing I have noticed is that cfstats is reporting a *LOT* of reads on the index, while
there's no way I'm firing that many queries:

                Local read count: 6670463
                Local read latency: 4.134 ms

Based on the application's logs, I would say there shouldn't have been more than 5000 queries
against the index since Cassandra was started on this node.

What may also be relevant is that during the upgrade to 2.1.7, nodes that were still on 2.1.5
were responding quickly (tested in cqlsh), while nodes that had already upgraded were slow.
Even when only one node was left to be upgraded, that node was fast with responding.


For completeness, the query that's slow is {{select * from the_table where id_2=?}}

  was:
After upgrading from 2.1.5 to 2.1.7, read performance has significantly slowed down for queries
that use indexes. A query that used to take under a second to execute now takes 8 seconds
or longer.

The table I am noticing this on is a fairly large table (possibly pushing Cassandra's limits
a bit, with around 1TB per node) with a single index.

{noformat}
CREATE TABLE the_table (
    id bigint PRIMARY KEY,
    id_2 bigint,
    field_1 ascii,
    field_2 blob, -- "Wide" row in that it's on average 15KB
    field_3 ascii,
    field_4 ascii,
    field_5 text,
    field_6 ascii
)  WITH compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
    AND compression = {'chunk_length_kb': '1024', 'sstable_compression': 'org.apache.cassandra.io.compress.DeflateCompressor'};
CREATE INDEX the_table_id_2_idx ON the_table (id_2);
{noformat}

Compaction status, according to nodetool's cfstats :

index:                SSTables in each level: [1714/4, 7, 0, 0, 0, 0, 0, 0, 0]
main:                SSTables in each level: [79/4, 1, 105/100, 1049/1000, 172, 0, 0, 0, 0]

It's had a few days to catch up, but the indexes don't seem to be catching up well. Do note
that the performance issue I'm reporting was also experienced when the main table had not
yet caught up. Queries fetching by the primary key have been equally fast before and after
the compaction catch-up.

One thing I have noticed is that cfstats is reporting a *LOT* of reads on the index, while
there's no way I'm firing that many queries:

                Local read count: 6670463
                Local read latency: 4.134 ms

Based on the application's logs, I would say there shouldn't have been more than 5000 queries
against the index since Cassandra was started on this node.

What may also be relevant is that during the upgrade to 2.1.7, nodes that were still on 2.1.5
were responding quickly (tested in cqlsh), while nodes that had already upgraded were slow.
Even when only one node was left to be upgraded, that node was fast with responding.


> Very bad read performance after 2.1.5->2.1.7 upgrade
> ----------------------------------------------------
>
>                 Key: CASSANDRA-9728
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9728
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Tom van der Woerdt
>
> After upgrading from 2.1.5 to 2.1.7, read performance has significantly slowed down for
queries that use indexes. A query that used to take under a second to execute now takes 8
seconds or longer.
> The table I am noticing this on is a fairly large table (possibly pushing Cassandra's
limits a bit, with around 1TB per node) with a single index.
> {noformat}
> CREATE TABLE the_table (
>     id bigint PRIMARY KEY,
>     id_2 bigint,
>     field_1 ascii,
>     field_2 blob, -- "Wide" column in that it's on average 15KB
>     field_3 ascii,
>     field_4 ascii,
>     field_5 text,
>     field_6 ascii
> )  WITH compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
>     AND compression = {'chunk_length_kb': '1024', 'sstable_compression': 'org.apache.cassandra.io.compress.DeflateCompressor'};
> CREATE INDEX the_table_id_2_idx ON the_table (id_2);
> {noformat}
> Compaction status, according to nodetool's cfstats :
> index:                SSTables in each level: [1714/4, 7, 0, 0, 0, 0, 0, 0, 0]
> main:                SSTables in each level: [79/4, 1, 105/100, 1049/1000, 172, 0, 0,
0, 0]
> It's had a few days to catch up, but the indexes don't seem to be catching up well. Do
note that the performance issue I'm reporting was also experienced when the main table had
not yet caught up. Queries fetching by the primary key have been equally fast before and after
the compaction catch-up.
> One thing I have noticed is that cfstats is reporting a *LOT* of reads on the index,
while there's no way I'm firing that many queries:
>                 Local read count: 6670463
>                 Local read latency: 4.134 ms
> Based on the application's logs, I would say there shouldn't have been more than 5000
queries against the index since Cassandra was started on this node.
> What may also be relevant is that during the upgrade to 2.1.7, nodes that were still
on 2.1.5 were responding quickly (tested in cqlsh), while nodes that had already upgraded
were slow. Even when only one node was left to be upgraded, that node was fast with responding.
> For completeness, the query that's slow is {{select * from the_table where id_2=?}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message