I don't have the additional hardware to try to isolate this issue atm, so I decided to push some code that performs 20% of reads directly from cassandra. The cache hit rate has gone up to about 88% now and it's still climbing, albeit slowly. There remains plenty of free cache space.

So far, the average time to multi_get those 20 rows is still hovering around 35-45ms.

I'll report back with more info as it comes in.

On Thu, Apr 1, 2010 at 12:06 AM, Cemal Dalar <cemal.dalar@gmail.com> wrote:
Hi James, 

I don't know how to get the below statistics data and calculate the access times (read/write in ms) in your previous mails. Can you explain a little? Iike to work on it also. 

CD


On Thu, Apr 1, 2010 at 4:15 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
On Wed, Mar 31, 2010 at 6:21 PM, James Golick <jamesgolick@gmail.com> wrote:
> Keyspace: ActivityFeed
>         Read Count: 699443
>         Read Latency: 16.11017477192566 ms.

>                 Column Family: Events
>                 Read Count: 232378
>                 Read Latency: 0.396 ms.
>                 Row cache capacity: 500000
>                 Row cache size: 62768
>                 Row cache hit rate: 0.007716049382716049

This says that

 - recent queries to Events are much faster than the lifetime average
for your Keyspace
 - even though you have almost no row cache hits (~1700 out of 232000 reads)

Not sure what to make of that, tbh.  If it were me I would try to
reproduce on a test machine w/o all that pesky live traffic confusing
things.

-Jonathan