incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Troubleshooting IO performance ?
Date Sun, 05 Jun 2011 14:54:17 GMT
You may be swapping.

http://spyced.blogspot.com/2010/01/linux-performance-basics.html
explains how to check this as well as how to see what threads are busy
in the Java process.

On Sat, Jun 4, 2011 at 5:34 PM, Philippe <watcherfr@gmail.com> wrote:
> Hello,
> I am evaluating using cassandra and I'm running into some strange IO
> behavior that I can't explain, I'd like some help/ideas to troubleshoot it.
> I am running a 1 node cluster with a keyspace consisting of two columns
> families, one of which has dozens of supercolumns itself containing dozens
> of columns.
> All in all, this is a couple gigabytes of data, 12GB on the hard drive.
> The hardware is pretty good : 16GB memory + RAID-0 SSD drives with LVM and
> an i5 processor (4 cores).
> Keyspace: xxxxxxxxxxxxxxxxxxx
>         Read Count: 460754852
>         Read Latency: 1.108205793092766 ms.
>         Write Count: 30620665
>         Write Latency: 0.01411020877567486 ms.
>         Pending Tasks: 0
>                 Column Family: xxxxxxxxxxxxxxxxxxxxxxxxxx
>                 SSTable count: 5
>                 Space used (live): 548700725
>                 Space used (total): 548700725
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 11
>                 Read Count: 2891192
>                 Read Latency: NaN ms.
>                 Write Count: 3157547
>                 Write Latency: NaN ms.
>                 Pending Tasks: 0
>                 Key cache capacity: 367396
>                 Key cache size: 367396
>                 Key cache hit rate: NaN
>                 Row cache capacity: 112683
>                 Row cache size: 112683
>                 Row cache hit rate: NaN
>                 Compacted row minimum size: 125
>                 Compacted row maximum size: 924
>                 Compacted row mean size: 172
>                 Column Family: yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy
>                 SSTable count: 7
>                 Space used (live): 8707538781
>                 Space used (total): 8707538781
>                 Memtable Columns Count: 0
>                 Memtable Data Size: 0
>                 Memtable Switch Count: 30
>                 Read Count: 457863660
>                 Read Latency: 2.381 ms.
>                 Write Count: 27463118
>                 Write Latency: NaN ms.
>                 Pending Tasks: 0
>                 Key cache capacity: 4518387
>                 Key cache size: 4518387
>                 Key cache hit rate: 0.9247881700850826
>                 Row cache capacity: 1349682
>                 Row cache size: 1349682
>                 Row cache hit rate: 0.39400533823415573
>                 Compacted row minimum size: 125
>                 Compacted row maximum size: 6866
>                 Compacted row mean size: 165
> My app makes a bunch of requests using a MultigetSuperSliceQuery for a set
> of keys, typically a couple dozen at most. It also selects a subset of the
> supercolumns. I am running 8 requests in parallel at most.
>
> Two days, I ran a 1.5 hour process that basically read every key. The server
> had no IOwaits and everything was humming along. However, right at the end
> of the process, there was a huge spike in IOs. I didn't think much of it.
> Today, after two days of inactivity, any query I run raises the IOs to 80%
> utilization of the SSD drives even though I'm running the same query over
> and over (no cache??)
> Any ideas on how to troubleshoot this, or better, how to solve this ?
> thanks
> Philippe



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message