Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1AFF310FDC for ; Wed, 4 Jun 2014 11:19:47 +0000 (UTC) Received: (qmail 47191 invoked by uid 500); 4 Jun 2014 11:19:44 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 47145 invoked by uid 500); 4 Jun 2014 11:19:44 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 47131 invoked by uid 99); 4 Jun 2014 11:19:44 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2014 11:19:44 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of belliottsmith@datastax.com designates 209.85.223.177 as permitted sender) Received: from [209.85.223.177] (HELO mail-ie0-f177.google.com) (209.85.223.177) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2014 11:19:41 +0000 Received: by mail-ie0-f177.google.com with SMTP id y20so7028966ier.22 for ; Wed, 04 Jun 2014 04:19:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=KGTkYI65j4whfAcnmS2aBvl9NW86eunkjttksAfV4Po=; b=ld17V7dLu6xz5519w/UIndFq3H4PFEUO/jwAJ1nToLHTKMZdOUQNlfmdlyf/HYmfGB BeDzwV33z0F+V0DYHpfRuqnq34RoO/HRW6dy9DUZKnJCeIwgd+/9HQDchfiL/aj7dAiw QOmy2usVKWSLdAFntHiev57Pa6aqQ7ZVGfaftWh9z++0LcaZzFPRxicxGZPjfYmsMbej CK/J7tbUE6srFk12hcOmh9ZMkbnPll9dAX//u8XJbCe1USp47+iE4XGKAJgY2Fl3ryTU YDOu4Qbxm+bSWQXg32cMknRkXRXFExTADmcAO/19S/Mic3fAUi93NtIZTP9YU3UbKO7F /w3Q== X-Gm-Message-State: ALoCoQmmqtOFtVfgFNwyDD19LEjyT27GE1DDvZgq6H5kHyuKpbFw43iWO3/di6Zjgi1NldQAmxqi MIME-Version: 1.0 X-Received: by 10.50.46.35 with SMTP id s3mr6125380igm.40.1401880756969; Wed, 04 Jun 2014 04:19:16 -0700 (PDT) Received: by 10.42.12.201 with HTTP; Wed, 4 Jun 2014 04:19:16 -0700 (PDT) In-Reply-To: <1401879841309.12539@dice.se> References: <1401875371141.25081@dice.se> <1401876241947.15837@dice.se> <1401879841309.12539@dice.se> Date: Wed, 4 Jun 2014 12:19:16 +0100 Message-ID: Subject: Re: memtable mem usage off by 10? From: Benedict Elliott Smith To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a113441bc89776204fb00cc56 X-Virus-Checked: Checked by ClamAV on apache.org --001a113441bc89776204fb00cc56 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Unfortunately it looks like the heap utilisation of memtables was not exposed in earlier versions, because they only maintained an estimate. The overhead scales linearly with the amount of data in your memtables (assuming the size of each cell is approx. constant). flush_largest_memtables_at is an independent setting to memtable_total_space_in_mb, and generally has little effect. Ordinarily sstable flushes are triggered by hitting the memtable_total_space_in_mb limit. I'm afraid I don't follow where your 3x comes from? On 4 June 2014 12:04, Idr=C3=A9n, Johan wrote: > Aha, ok. Thanks. > > > Trying to understand what my cluster is doing: > > > cassandra.db.memtable_data_size only gets me the actual data but not the > memtable heap memory usage. Is there a way to check for heap memory usage= ? > > > I would expect to hit the flush_largest_memtables_at value, and this > would be what causes the memtable flush to sstable then? By default 0.75? > > > Then I would expect the amount of memory to be used to be maximum ~3x of > what I was seeing when I hadn't set memtable_total_space_in_mb (1/4 by > default, max 3/4 before a flush), instead of close to 10x (250mb vs 2gb). > > > This is of course assuming that the overhead scales linearly with the > amount of data in my table, we're using one table with three cells in thi= s > case. If it hardly increases at all, then I'll give up I guess :) > > At least until 2.1.0 comes out and I can compare. > > > BR > > Johan > > > ------------------------------ > *From:* Benedict Elliott Smith > *Sent:* Wednesday, June 4, 2014 12:33 PM > > *To:* user@cassandra.apache.org > *Subject:* Re: memtable mem usage off by 10? > > These measurements tell you the amount of user data stored in the > memtables, not the amount of heap used to store it, so the same applies. > > > On 4 June 2014 11:04, Idr=C3=A9n, Johan wrote: > >> I'm not measuring memtable size by looking at the sstables on disk, no. >> I'm looking through the JMX data. So I would believe (or hope) that I'm >> getting relevant data. >> >> >> If I have a heap of 10GB and set the memtable usage to 20GB, I would >> expect to hit other problems, but I'm not seeing memory usage over 10GB = for >> the heap, and the machine (which has ~30gb of memory) is showing ~10GB >> free, with ~12GB used by cassandra, the rest in caches. >> >> >> Reading 8k rows/s, writing 2k rows/s on a 3 node cluster. So it's not >> idling. >> >> >> BR >> >> Johan >> >> >> ------------------------------ >> *From:* Benedict Elliott Smith >> *Sent:* Wednesday, June 4, 2014 11:56 AM >> *To:* user@cassandra.apache.org >> *Subject:* Re: memtable mem usage off by 10? >> >> If you are storing small values in your columns, the object overhead >> is very substantial. So what is 400Mb on disk may well be 4Gb in memtabl= es, >> so if you are measuring the memtable size by the resulting sstable size, >> you are not getting an accurate picture. This overhead has been reduced = by >> about 90% in the upcoming 2.1 release, through tickets 6271 >> , 6689 >> and 6694 >> . >> >> >> On 4 June 2014 10:49, Idr=C3=A9n, Johan wrote: >> >>> Hi, >>> >>> >>> I'm seeing some strange behavior of the memtables, both in 1.2.13 and >>> 2.0.7, basically it looks like it's using 10x less memory than it shoul= d >>> based on the documentation and options. >>> >>> >>> 10GB heap for both clusters. >>> >>> 1.2.x should use 1/3 of the heap for memtables, but it uses max ~300mb >>> before flushing >>> >>> >>> 2.0.7, same but 1/4 and ~250mb >>> >>> >>> In the 2.0.7 cluster I set the memtable_total_space_in_mb to 4096, >>> which then allowed cassandra to use up to ~400mb for memtables... >>> >>> >>> I'm now running with 20480 for memtable_total_space_in_mb and >>> cassandra is using ~2GB for memtables. >>> >>> >>> Soo, off by 10 somewhere? Has anyone else seen this? Can't find a JIRA >>> for any bug connected to this. >>> >>> java 1.7.0_55, JNA 4.1.0 (for the 2.0 cluster) >>> >>> >>> BR >>> >>> Johan >>> >> >> > --001a113441bc89776204fb00cc56 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Unfortunately it looks like the heap utilisation of memtab= les was not exposed in earlier versions, because they only maintained an es= timate.

The overhead scales linearly with the amount of = data in your memtables (assuming the size of each cell is approx. constant)= .=C2=A0

flush_largest_memtables_at is an independent setting to= memtable_total_space_in_mb, and generally has little effect. Ordinarily ss= table flushes are triggered by hitting the memtable_total_space_in_mb limit= . I'm afraid I don't follow where your 3x comes from?


O= n 4 June 2014 12:04, Idr=C3=A9n, Johan <Johan.Idren@dice.se> wrote:

Aha, ok. Thanks.


Trying to understand what my cluster is doing:


cassandra.db.memtable_data_size=C2=A0only gets me the actual data but not the = memtable heap memory usage. Is there a way to check for heap memory usage?<= /span>


I would expect to hit the flush_largest_m= emtables_at value, and this would be what causes the memtable=C2=A0flush to= sstable=C2=A0then? By default 0.75?


Then I would expect the amount of memory = to be used to be maximum=C2=A0~3x of what I was seeing when I hadn't se= t memtable_total_space_in_mb=C2=A0(1/4 by default, max=C2=A03/4 before a fl= ush), instead of close to 10x (250mb vs 2gb).


This is of course assuming that the overhead scales linearly with the amoun= t of data in my table, we're using one table with three cells in this c= ase. If it hardly increases at all, then I'll give up I guess :)=

At least until 2.1.0 comes out and I can = compare.


BR

Johan


From: Benedict Elliott Smith <= ;belliottsm= ith@datastax.com>
Sent: Wednesday, June 4, 2014 12:33 PM
<= br> To: u= ser@cassandra.apache.org
Subject: Re: memtable mem usage off by 10?
=C2=A0
These measurements tell you the amount of user data stored= in the memtables, not the amount of heap used to store it, so the same app= lies.


On 4 June 2014 11:04, Idr=C3=A9n, Johan <Joha= n.Idren@dice.se> wrote:

I'm not measuring memtable size by looking at the sstables on disk, = no. I'm looking through the JMX data. So I would believe (or hope) that= I'm getting relevant data.=C2=A0


If I have a heap of 10GB and set the memtable usage to 20GB, I would exp= ect to hit other problems, but I'm not seeing memory usage over 10GB fo= r the heap, and the machine (which has ~30gb of memory) is showing ~10GB fr= ee, with ~12GB used by cassandra, the rest in caches.


Reading 8k rows/s, writing 2k rows/s on a 3 node cluster. So it's no= t idling.


BR

Johan


From: Benedict Elliott Smith <belliottsmith@datastax.com= >
Sent: Wednesday, June 4, 2014 11:56 AM
To: u= ser@cassandra.apache.org
Subject: Re: memtable mem usage off by 10?
=C2=A0
If you are storing small values in your columns, the objec= t overhead is very substantial. So what is 400Mb on disk may well be 4Gb in= memtables, so if you are measuring the memtable size by the resulting ssta= ble size, you are not getting an accurate picture. This overhead has been reduced by about 90% in the upcoming 2.1 r= elease, through tickets 6271, 6689=C2=A0and=C2=A06694.


On 4 June 2014 10:49, Idr=C3=A9n, Johan <Joha= n.Idren@dice.se> wrote:

Hi,


I'm seeing some strange behavior of the memtables, both in 1.2.13 an= d 2.0.7, basically it looks like it's using 10x less memory than it sho= uld based on the documentation and options.


10GB heap for both clusters.

1.2.x should use 1/3 of the heap for memtables, but it uses max ~300mb b= efore flushing


2.0.7, same but 1/4 and ~250mb


In the 2.0.7 cluster I set the=C2=A0memtable_total_space_in_mb to 4096, = which then allowed cassandra to use up to ~400mb for memtables...


I'm now running with 20480 for=C2=A0memtable_total_space_in_mb and c= assandra is using ~2GB for memtables.


Soo, off by 10 somewhere? Has anyone else seen this? Can't find a JI= RA for any bug connected to this.

java=C2=A01.7.0_55, JNA 4.1.0 (for the 2.0 cluster)


BR

Johan




--001a113441bc89776204fb00cc56--