cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Bligh (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-7103) Very poor performance with simple setup
Date Tue, 29 Apr 2014 12:06:14 GMT


Martin Bligh commented on CASSANDRA-7103:

OK, that explains some stuff ... will take it to user list as you suggest. I was under the
impression that the row key was the whole of the primary key, not just the partition key,
as described here, for instance:
"The first component of a table's primary key is the partition key; within a partition, rows
are clustered by the remaining columns of the PK."

In which case, that description seems very misleading

> Very poor performance with simple setup
> ---------------------------------------
>                 Key: CASSANDRA-7103
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Fedora 19 (also happens on Ubuntu), Cassandra 2.0.7. dsc standard
>            Reporter: Martin Bligh
> Single node (this is just development, 32GB 20 core server), single disk array.
> Create the following table:
> {code}
>   time_order bigint,
>   time_start bigint,
>   ack_us map<int, int>,
>   gc_strategy map<text, int>,
>   gc_strategy_symbol map<text, int>,
>   gc_symbol map<text, int>,
>   ge_strategy map<text, int>,
>   ge_strategy_symbol map<text, int>,
>   ge_symbol map<text, int>,
>   go_strategy map<text, int>,
>   go_strategy_symbol map<text, int>,
>   go_symbol map<text, int>,
>   message_type map<text, int>,
>   PRIMARY KEY (time_order, time_start)
> ) WITH
>   bloom_filter_fp_chance=0.010000 AND
>   caching='KEYS_ONLY' AND
>   comment='' AND
>   dclocal_read_repair_chance=0.000000 AND
>   gc_grace_seconds=864000 AND
>   index_interval=128 AND
>   read_repair_chance=0.100000 AND
>   replicate_on_write='true' AND
>   populate_io_cache_on_flush='false' AND
>   default_time_to_live=0 AND
>   speculative_retry='99.0PERCENTILE' AND
>   memtable_flush_period_in_ms=0 AND
>   compaction={'class': 'SizeTieredCompactionStrategy'} AND
>   compression={};
> {code}
> Now I just insert data into it (using python driver, async insert, prepared insert statement).
Each row only fills out one of the gc_*, go_*, or ge_* columns, and there's something like
20-100 entries per map column, occasionally 1000, but it's nothing huge. 
> First run 685 inserts in 1.004860 seconds (681.687053 Operations/s).
> OK, not great, but that's fine.
> Now throw 50,000 rows at it.
> Now run the first run again, and it takes 53s to do the same insert of 685 rows - I'm
getting about 10 rows per second. 
> It's not IO bound - "iostat 1" shows quiescent for 9 seconds, then ~640KB write, then
sleeps again - seems like the fflush sync.
> Run "nodetool flush" and performance goes back to as before!!!!
> Not sure why this gets so slow - I think it just builds huge commit logs and memtables,
but never writes out to the data/ directory with sstables because I only have one table? That
doesn't seem like a good situation. 
> Worse ... if you let the python driver just throw stuff at it async (I think this allows
up to 128 request if I understand the underlying protocol, then it gets so slow that a single
write takes over 10s and times out). Seems to be some sort of synchronization problem in Java
... if I limit the concurrent async requests to the left column below, I get the number of
seconds elapsed on the right:
> 1: 103 seconds
> 2: 63 seconds
> 8: 53 seconds
> 16: 53 seconds
> 32: 66 seconds
> 64: so slow it explodes in timeouts on write (over 10s each).
> I guess there's some thundering herd type locking issue in whatever Java primitive you
are using to lock concurrent access to a single table. I know some of the Java concurrent.*
stuff has this issue. So for the other tests above, I was limiting async writes to 16 pending.

This message was sent by Atlassian JIRA

View raw message