We are using Leveled Compaction with Cassandra 1.2.5. Our sstable size is 100M. On each node,
we have anywhere from 700+ to 800+ sstables (for all levels). The bloom_filter_fp_chance is set at 0.000744.

  For read requests that ask for existing row records, the latency is great, mostly under 20 milliseconds with key cache and row cache set. For read requests that ask for non-existing row records (not because of deletes, but rather, have never been in the system to start with), the latency is running into hundreds of milliseconds and sometimes seconds.

  Just wonder if anyone has come across this before and has some pointers on how to reduce the latency in this case.