incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Plowe <eric.pl...@gmail.com>
Subject Noticing really high read latency
Date Wed, 05 Mar 2014 06:44:39 GMT
Background info:

6 node cluster.
24 gigs of ram per machine
8 gigs of ram dedicated to c*
4 4 core cpu's
2 250 gig SSD's raid 0
Running c* 1.2.6

The CF is configured as followed

CREATE TABLE behaviors (
  uid text,
  buid int,
  name text,
  expires text,
  value text,
  PRIMARY KEY (uid, buid, name)
) WITH
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  compaction={'sstable_size_in_mb': '160', 'class':
'LeveledCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

I am noticing that the read latency is very high considering when I look at
the output of nodetool cfstats.

This is the example output of one of the nodes:

  Column Family: behaviors
                SSTable count: 2
                SSTables in each level: [1, 1, 0, 0, 0, 0, 0, 0, 0]
                Space used (live): 171496198
                Space used (total): 171496591
                Number of Keys (estimate): 1153664
                Memtable Columns Count: 14445
                Memtable Data Size: 1048576
                Memtable Switch Count: 1
                Read Count: 1894
                Read Latency: 0.497 ms.
                Write Count: 7169
                Write Latency: 0.041 ms.
                Pending Tasks: 0
                Bloom Filter False Positives: 4
                Bloom Filter False Ratio: 0.00862
                Bloom Filter Space Used: 3533152
                Compacted row minimum size: 125
                Compacted row maximum size: 9887
                Compacted row mean size: 365

The write latency is awesome, but the read latency, not so much. The output
of iostat doesn't show anything out of the ordinary. The cpu utilization is
between 1% to 5%.

All read queries are issued with a CL of ONE. We always include "WHERE uid
= '<somevalue>'" for the queries.

If there is any more info I can provide, please let me know. At this point
in time, I am a bit stumped.

Regards,

Eric Plowe

Mime
View raw message