Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of rcoli@digg.com designates
 209.85.214.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1298505882042-6058435.post@n2.nabble.com>
References: <1298505882042-6058435.post@n2.nabble.com>
Date: Wed, 23 Feb 2011 16:21:43 -0800
Message-ID: <AANLkTi=Aoi3MKR9NEM3eLr-QfkqrdryJZ=-mnEp59R+m@mail.gmail.com>
Subject: Re: How come key cache increases speed by x4?
From: Robert Coli <rcoli@digg.com>
To: user@cassandra.apache.org
Cc: buddhasystem <potekhin@bnl.gov>, cassandra-user@incubator.apache.org
Content-Type: text/plain; charset=ISO-8859-1

On Wed, Feb 23, 2011 at 4:04 PM, buddhasystem <potekhin@bnl.gov> wrote:

> Well I know the cache is there for a reason, I just can't explain the factor
> of 4 when I run my queries on a hot vs cold cache. My queries are actually a
> chain of one on an inverted index, which produces a tuple of keys to be used
> in the "main" query. The inverted index query should be downright trivial.
>
> I see the turnaround time per row go down to 1 ms from 4 ms. Am I missing
> something? Why such a large factor?

(simplified for discussion purposes, not necessarily exhaustive
description of.. )

Path in the cold key cache case :

a) check all bloom filters, 1 per sstable in the CF, which is in memory
b) read the index file (not in memory) and traverse index for every
sstable which returns positive in a)
c) read the actual data file once for every sstable

Path in the hot key cache case :

a) read list of filenames and offsets from key cache
b) read the actual data file

You will notice that the former involves a lot more seeking than the
latter, especially if you have "many" sstables. This seeking almost
certainly is the cause of your observed difference. If you graph I/O
throughput in the two different cases, you will almost certainly see
yourself doing more (slow) I/O in the cold cache case. Memory spent on
key cache is usually relatively well spent, for this reason.

=Rob