Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of omidaladini@gmail.com
 designates 209.85.160.44 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <1745B02C-7AA2-4204-A220-BAABD75A6856@thelastpickle.com>
References: 
 <CAH-t0tZ1vfgBFo=0g7i0cU7f=1bd5a9yTfHqy6pfjN0oVoU0Tg@mail.gmail.com>
 <1745B02C-7AA2-4204-A220-BAABD75A6856@thelastpickle.com>
From: Omid Aladini <omidaladini@gmail.com>
Date: Fri, 11 May 2012 18:10:41 +0200
Message-ID: 
 <CAH-t0tYiPExToW1yNU4zCB3YLvHYxHMtjYw=m=1XbWB5cUnnYw@mail.gmail.com>
Subject: Re: Losing key cache on restart
To: user@cassandra.apache.org
Content-Type: text/plain; charset=UTF-8

Hey,

Sorry for the late response.

On Wed, Apr 25, 2012 at 1:36 AM, aaron morton <aaron@thelastpickle.com> wrote:
> - Cassandra log reports 12,955,585 of them have been saved on the last save
> events.
>
> Has their been much activity between saves ?

For testing I set the key_cache_save_period to a short period of 10
minutes. How would this affect the result? Does saving the cache also
prunes duplicate in-memory elements?

> Nothing jumps out. There is a setting for the max entries to store, but this
> only applies to the row cache. Can you reproduce issue in a dev environment
> ?

So far I haven't been able to reproduce this in development environment.

> When running the key cache holds keys of the form <SSTable, DecoratedKey> so
> there is an entry for each SSTable the key appears in. When saved only the
> DecoratedKey's are stored, and the key cache is rebuilt on startup when
> iterating over the index files. e.g. ig you have 12 entries in the keycache,
> it may only be 4 unique keys and that is all that is written when saving the
> cache.

If you have 12 entries for the same key on 12 sstables, this means
your data is spread across 12 sstables, how could cassandra
deduplicate them to 4?

I'll try to reproduce/debug it as well.

Thanks,
Omid

>
> From a quick look at the code it looks like the code is writing all
> DecoratedKeys , not just the unique ones. This may be mucking up the
> reported numbers, I'll take a look later.
>
> If you can reproduce it simply it would help.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 25/04/2012, at 1:11 AM, Omid Aladini wrote:
>
> Hi,
>
> I'm experiencing losing a part of key cache on restart on Cassandra 1.0.7.
> For example:
>
> - cfstats reports key cache size of 13,040,502 with capacity of 15,000,000.
> - Cassandra log reports 12,955,585 of them have been saved on the last save
> events.
> - On restart Cassandra reads saved cache.
> - cfstats reports key cache size of only 2,833,586 with correct capacity of
> 15,000,000.
>
> There is no sign that the cache size is reduced due to memory pressure. The
> key cache capacity is set manually via cassandra-cli.
>
> Has anyone else encountered this problem or is it a known issue?
>
> Thanks,
> Omid
>
>