Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Message-ID: <564C6484.2040502@stickyads.tv>
Date: Wed, 18 Nov 2015 12:44:04 +0100
From: Antoine Bonavita <antoine@stickyads.tv>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:17.0) Gecko/20130805 Thunderbird/17.0.8
MIME-Version: 1.0
To: user@cassandra.apache.org, Robert Coli <rcoli@eventbrite.com>,
 sebastian.estevez@datastax.com
Subject: Re: Help diagnosing performance issue
References: <56499C10.8030605@stickyads.tv> <564B7312.3030307@stickyads.tv>
 <CACCmCN_AxsvAOpDP9GNRwVOAH7c4h8dB8bN9_zJQyC1FqAe17w@mail.gmail.com>
 <CAEDUwd2Ux=skqSQhvAGvUX8vQNKjOGCMTTGmMs8O6ev5rgYxRg@mail.gmail.com>
In-Reply-To: 
 <CAEDUwd2Ux=skqSQhvAGvUX8vQNKjOGCMTTGmMs8O6ev5rgYxRg@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Sebastian, Robet,

First, a big thank you to both of you for your help.

It looks like you were right. I used pcstat (awesome tool, thanks for 
that as well) and it appears some files I would not expect to be in 
cache actually are. Here is a sample of my output (edited for 
convenience, adding the file timestamp from the OS):

* 
/var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5951-big-Data.db 
- 000.619 % - Nov 16 12:25
* 
/var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5954-big-Data.db 
- 000.681 % - Nov 16 13:44
* 
/var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5955-big-Data.db 
-  000.610 % - Nov 16 14:11
* 
/var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5956-big-Data.db 
- 015.621 % - Nov 16 14:26
* 
/var/lib/cassandra/data/views/views-451e4d8061ef11e5896f091196a360a0/la-5957-big-Data.db 
- 015.558 % - Nov 16 14:50

The SSTables that come before are all at about 0% and the ones that come 
after it are all at about 15%.

As you can see the first SSTable at 15% date back from 24h. Given my 
application I'm pretty sure those are not from the reads (reads of data 
older than 1h is definitely under 0.1% of reads). Could it be that 
compaction is putting those in cache constantly ?
If so, then I'm probably confused on the meaning/effect of 
max_sstable_age_days (set at 10 in my case) and base_time_seconds (not 
set in my case so the default of 3600 applies). I would not expect any 
compaction to happen beyond the first hour and the 10 days is here to 
make sure data still gets expired and SSTables removed (thus releasing 
disk space). I don't see where the 24h come from.
If you guys can shed some light on this, it would be awesome. I'm sure I 
got something wrong.

Regarding the heap configuration, both are very similar:
* 32G machine: -Xms8049M -Xmx8049M -Xmn800M
* 64G machine: -Xms8192M -Xmx8192M -Xmn1200M
I think we can rule that out.

Thanks again for you help, I truly appreciate it.

A.

On 11/17/2015 08:48 PM, Robert Coli wrote:
> On Tue, Nov 17, 2015 at 11:08 AM, Sebastian Estevez
> <sebastian.estevez@datastax.com <mailto:sebastian.estevez@datastax.com>>
> wrote:
>
>     You're sstables are probably falling out of page cache on the
>     smaller nodes and your slow disks are killing your latencies.
>
>
> +1 most likely.
>
> Are the heaps the same size on both machines?
>
> =Rob

-- 
Antoine Bonavita (antoine@stickyads.tv) - CTO StickyADS.tv
Tel: +33 6 34 33 47 36/+33 9 50 68 21 32
NEW YORK | LONDON | HAMBURG | PARIS | MONTPELLIER | MILAN | MADRID