All,

 I've done a bit more homework, and I continue to see long 200ms to 300ms read times for some keys.

Test Setup

EC2 M1Large sending requests to a 5 node C* cluster also in EC2, also all M1Large. RF=3. ReadConsistency = ONE. I'm using pycassa from python for all communication.

Data Model

One column family with tens of millions of rows. The number of columns per row varies between 0 and 1440 (per minute records). The values are all ints. All data stored on EBS volumes. Total load per node is ~110GB.

According to VMstat I'm not swapping at all.

Highest %Util I see
Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvdf              0.00  2788.00   17.00  267.50  1168.00 23020.00    85.02    32.37  107.73   1.22  34.60

A more average profile I see is:

Device:         rrqm/s   wrqm/s     r/s     w/s   rsec/s   wsec/s avgrq-sz avgqu-sz   await  svctm  %util
xvdf              0.00     0.00   21.00    0.00  1288.00     0.00    61.33     0.37   18.38   9.43  19.80

QUESTION

Where should I look next? I'd love to get a profile of exactly where cassandra is spending its time on a per call basis.

Thanks in advance,

Ian