Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9510F1183F for ; Wed, 4 Jun 2014 15:18:58 +0000 (UTC) Received: (qmail 24285 invoked by uid 500); 4 Jun 2014 15:18:55 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 24250 invoked by uid 500); 4 Jun 2014 15:18:55 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 24242 invoked by uid 99); 4 Jun 2014 15:18:55 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2014 15:18:55 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of belliottsmith@datastax.com designates 209.85.223.171 as permitted sender) Received: from [209.85.223.171] (HELO mail-ie0-f171.google.com) (209.85.223.171) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Jun 2014 15:18:51 +0000 Received: by mail-ie0-f171.google.com with SMTP id to1so7430950ieb.30 for ; Wed, 04 Jun 2014 08:18:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=d2xgl/XFHorWMu3E4Pr5cvqjqUgGS9hrw2QygyJwPGc=; b=FumBaOBUll3P5avdKkWHFfAdvxmJ3z49hJ4etMI+bN1Oy/+oXr9Kz4+oGBOvVY3381 Ln8m1asbsmRSosnX+rB0/kKBmC6j/HdZ2uEkyRFzfEfLQkC2p5Y8T4fTmGizEBmy63vu g1SOmtMvY48DtSuMGgQ9Lbd+gUGrebtYG+hyHHbDIJezBZtWCYRveupkNQAw4Sitbnw3 T+V6Gzwb9Q8RAdeuCXkd2muI7uT8rlo6wcZ+SRJ95Rb6+9oDhUrKRgcEHY8Puo64IgVM o0BsQfPaJoEYpMnLhtSgqUAZPe3RuCXrjp+D9/FSR2ETyi67aXcIu9yXINBch01EpwYU iUzA== X-Gm-Message-State: ALoCoQlE0zSgLBEwAB7Lzq3xG+2tKpmZ/A59iOwHuUCDRlLO4HyTOGXaoCD2PEtkXqfQAZfh7udE MIME-Version: 1.0 X-Received: by 10.50.80.50 with SMTP id o18mr8399297igx.40.1401895111052; Wed, 04 Jun 2014 08:18:31 -0700 (PDT) Received: by 10.42.12.201 with HTTP; Wed, 4 Jun 2014 08:18:30 -0700 (PDT) In-Reply-To: References: <1401875371141.25081@dice.se> <1401876241947.15837@dice.se> <1401879841309.12539@dice.se> <1401882919755.3901@dice.se> <1401884768496.56712@dice.se> <3FD7F8AA2F44469481822F854E288BD3@JackKrupansky14> Date: Wed, 4 Jun 2014 16:18:30 +0100 Message-ID: Subject: Re: memtable mem usage off by 10? From: Benedict Elliott Smith To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=089e0149cef61b72e804fb04243d X-Virus-Checked: Checked by ClamAV on apache.org --089e0149cef61b72e804fb04243d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable In that case I would assume the problem is that for some reason JAMM is failing to load, and so the liveRatio it would ordinarily calculate is defaulting to 10 - are you using the bundled cassandra launch scripts? On 4 June 2014 15:51, Idr=C3=A9n, Johan wrote: > I wasn=E2=80=99t supplying it, I was assuming it was using the default. = It does > not exist in my config file. Sorry for the confusion. > > > > From: Benedict Elliott Smith > Reply-To: "user@cassandra.apache.org" > Date: Wednesday 4 June 2014 16:36 > To: "user@cassandra.apache.org" > > Subject: Re: memtable mem usage off by 10? > > Oh, well ok that explains why I'm not seeing a flush at 750MB. Sorry, >> I was going by the documentation. It claims that the property is around = in >> 2.0. > > But something else is wrong, as Cassandra will crash if you supply an > invalid property, implying it's not sourcing the config file you're using= . > I'm afraid I don't have the context for why it was removed, but it > happened as part of the 2.0 release. > >> > > On 4 June 2014 13:59, Jack Krupansky wrote: > >> Yeah, it is in the doc: >> >> http://www.datastax.com/documentation/cassandra/2.0/cassandra/configurat= ion/configCassandra_yaml_r.html >> >> And I don=E2=80=99t find a Jira issue mentioning it being removed, so...= what=E2=80=99s >> the full story there?! >> >> -- Jack Krupansky >> >> *From:* Idr=C3=A9n, Johan >> *Sent:* Wednesday, June 4, 2014 8:26 AM >> *To:* user@cassandra.apache.org >> *Subject:* RE: memtable mem usage off by 10? >> >> >> Oh, well ok that explains why I'm not seeing a flush at 750MB. Sorry, I >> was going by the documentation. It claims that the property is around in >> 2.0. >> >> >> >> If we skip that, part of my reply still makes sense: >> >> >> >> Having memtable_total_size_in_mb set to 20480, memtables are flushed at = a >> reported value of ~2GB. >> >> >> >> With a constant overhead of ~10x, as suggested, this would mean that it >> used 20GB, which is 2x the size of the heap. >> >> >> >> That shouldn't work. According to the OS, cassandra doesn't use more tha= n >> ~11-12GB. >> >> >> ------------------------------ >> *From:* Benedict Elliott Smith >> *Sent:* Wednesday, June 4, 2014 2:07 PM >> *To:* user@cassandra.apache.org >> *Subject:* Re: memtable mem usage off by 10? >> >> I'm confused: there is no flush_largest_memtables_at property in C* 2.0= ? >> >> >> On 4 June 2014 12:55, Idr=C3=A9n, Johan wrote: >> >>> Ok, so the overhead is a constant modifier, right. >>> >>> >>> >>> The 3x I arrived at with the following assumptions: >>> >>> >>> >>> heap is 10GB >>> >>> Default memory for memtable usage is 1/4 of heap in c* 2.0 >>> max memory used for memtables is 2,5GB (10/4) >>> >>> flush_largest_memtables_at is 0.75 >>> >>> flush largest memtables when memtables use 7,5GB (3/4 of heap, 3x of th= e >>> default) >>> >>> >>> >>> With an overhead of 10x, it makes sense that my memtable is flushed whe= n >>> the jmx data says it is at ~250MB, ie 2,5GB, ie 1/4 of the heap >>> >>> >>> >>> After I've set the memtable_total_size_in_mb to a value larger than >>> 7,5GB, it should still not go over 7,5GB on account of >>> flush_largest_memtables_at, 3/4 the heap >>> >>> >>> >>> So I would expect to see memtables flushed to disk after they're being >>> reportedly at around 750MB. >>> >>> >>> >>> Having memtable_total_size_in_mb set to 20480, memtables are flushed at >>> a reported value of ~2GB. >>> >>> >>> >>> With a constant overhead, this would mean that it used 20GB, which is 2= x >>> the size of the heap, instead of 3/4 of the heap as it should be if >>> flush_largest_memtables_at was being respected. >>> >>> >>> >>> This shouldn't be possible. >>> >>> >>> ------------------------------ >>> *From:* Benedict Elliott Smith >>> *Sent:* Wednesday, June 4, 2014 1:19 PM >>> >>> *To:* user@cassandra.apache.org >>> *Subject:* Re: memtable mem usage off by 10? >>> >>> Unfortunately it looks like the heap utilisation of memtables was not >>> exposed in earlier versions, because they only maintained an estimate. >>> >>> The overhead scales linearly with the amount of data in your memtables >>> (assuming the size of each cell is approx. constant). >>> >>> flush_largest_memtables_at is an independent setting to >>> memtable_total_space_in_mb, and generally has little effect. Ordinarily >>> sstable flushes are triggered by hitting the memtable_total_space_in_mb >>> limit. I'm afraid I don't follow where your 3x comes from? >>> >>> >>> On 4 June 2014 12:04, Idr=C3=A9n, Johan wrote: >>> >>>> Aha, ok. Thanks. >>>> >>>> >>>> >>>> Trying to understand what my cluster is doing: >>>> >>>> >>>> >>>> cassandra.db.memtable_data_size only gets me the actual data but not >>>> the memtable heap memory usage. Is there a way to check for heap memor= y >>>> usage? >>>> >>>> >>>> I would expect to hit the flush_largest_memtables_at value, and this >>>> would be what causes the memtable flush to sstable then? By default 0.= 75? >>>> >>>> >>>> Then I would expect the amount of memory to be used to be maximum ~3x >>>> of what I was seeing when I hadn't set memtable_total_space_in_mb (1/4= by >>>> default, max 3/4 before a flush), instead of close to 10x (250mb vs 2g= b). >>>> >>>> >>>> This is of course assuming that the overhead scales linearly with the >>>> amount of data in my table, we're using one table with three cells in = this >>>> case. If it hardly increases at all, then I'll give up I guess :) >>>> >>>> At least until 2.1.0 comes out and I can compare. >>>> >>>> >>>> BR >>>> >>>> Johan >>>> >>>> >>>> ------------------------------ >>>> *From:* Benedict Elliott Smith >>>> *Sent:* Wednesday, June 4, 2014 12:33 PM >>>> >>>> *To:* user@cassandra.apache.org >>>> *Subject:* Re: memtable mem usage off by 10? >>>> >>>> These measurements tell you the amount of user data stored in the >>>> memtables, not the amount of heap used to store it, so the same applie= s. >>>> >>>> >>>> On 4 June 2014 11:04, Idr=C3=A9n, Johan wrote: >>>> >>>>> I'm not measuring memtable size by looking at the sstables on disk, >>>>> no. I'm looking through the JMX data. So I would believe (or hope) th= at I'm >>>>> getting relevant data. >>>>> >>>>> >>>>> >>>>> If I have a heap of 10GB and set the memtable usage to 20GB, I would >>>>> expect to hit other problems, but I'm not seeing memory usage over 10= GB for >>>>> the heap, and the machine (which has ~30gb of memory) is showing ~10G= B >>>>> free, with ~12GB used by cassandra, the rest in caches. >>>>> >>>>> >>>>> >>>>> Reading 8k rows/s, writing 2k rows/s on a 3 node cluster. So it's not >>>>> idling. >>>>> >>>>> >>>>> >>>>> BR >>>>> >>>>> Johan >>>>> >>>>> >>>>> ------------------------------ >>>>> *From:* Benedict Elliott Smith >>>>> *Sent:* Wednesday, June 4, 2014 11:56 AM >>>>> *To:* user@cassandra.apache.org >>>>> *Subject:* Re: memtable mem usage off by 10? >>>>> >>>>> If you are storing small values in your columns, the object >>>>> overhead is very substantial. So what is 400Mb on disk may well be 4G= b in >>>>> memtables, so if you are measuring the memtable size by the resulting >>>>> sstable size, you are not getting an accurate picture. This overhead = has >>>>> been reduced by about 90% in the upcoming 2.1 release, through ticket= s >>>>> 6271 , 6689 >>>>> and 6694 >>>>> . >>>>> >>>>> >>>>> On 4 June 2014 10:49, Idr=C3=A9n, Johan wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> >>>>>> >>>>>> I'm seeing some strange behavior of the memtables, both in 1.2.13 an= d >>>>>> 2.0.7, basically it looks like it's using 10x less memory than it sh= ould >>>>>> based on the documentation and options. >>>>>> >>>>>> >>>>>> >>>>>> 10GB heap for both clusters. >>>>>> >>>>>> 1.2.x should use 1/3 of the heap for memtables, but it uses max >>>>>> ~300mb before flushing >>>>>> >>>>>> >>>>>> >>>>>> 2.0.7, same but 1/4 and ~250mb >>>>>> >>>>>> >>>>>> >>>>>> In the 2.0.7 cluster I set the memtable_total_space_in_mb to 4096, >>>>>> which then allowed cassandra to use up to ~400mb for memtables... >>>>>> >>>>>> >>>>>> >>>>>> I'm now running with 20480 for memtable_total_space_in_mb and >>>>>> cassandra is using ~2GB for memtables. >>>>>> >>>>>> >>>>>> >>>>>> Soo, off by 10 somewhere? Has anyone else seen this? Can't find a >>>>>> JIRA for any bug connected to this. >>>>>> >>>>>> java 1.7.0_55, JNA 4.1.0 (for the 2.0 cluster) >>>>>> >>>>>> >>>>>> >>>>>> BR >>>>>> >>>>>> Johan >>>>>> >>>>> >>>>> >>>> >>>> >>> >>> >> >> > > --089e0149cef61b72e804fb04243d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
In that case I would assume the problem is that for some r= eason JAMM is failing to load, and so the liveRatio it would ordinarily cal= culate is defaulting to 10 - are you using the bundled cassandra launch scr= ipts?=C2=A0


On 4 June 201= 4 15:51, Idr=C3=A9n, Johan <Johan.Idren@dice.se> wrote:
I wasn=E2=80=99t supplying it, I was assuming it was using the default= . It does not exist in my config file. Sorry for the confusion.



From: Benedict Elliott Smith <belliottsmith= @datastax.com>
Reply-To: "user@cassandra.apache.org= " <user@cassandra.apache.org>
Date: Wednesday 4 June 2014 16:36 To: "user@cassandra.apache.org" &= lt;user@cass= andra.apache.org>

Subject: Re: memtable mem usage off= by 10?

Oh, well ok that explains why I'm not seeing a flus= h at 750MB. Sorry, I was going by the documentation. It claims that the pro= perty is around in 2.0.

But something else is wrong, as Cassandra will crash if you = supply an invalid property, implying it's not sourcing the config file = you're using.

I'm afraid I don't have the context for why it was removed, bu= t it happened as part of the 2.0 release.


On 4 June 2014 13:59, Jack Krupansky <j= ack@basetechnology.com> wrote:
Yeah, it is in the doc:
=C2=A0
And I don=E2=80=99t find a Jira issue mentioning it being removed, so.= .. what=E2=80=99s the full story there?!
=C2=A0
-= - Jack Krupansky
=C2=A0
Sent: Wednesday, June 4, 2014 8:26 AM
Subject: RE: memtable mem usage off by 10?
=C2=A0

Oh, well ok that explains why I'm not seeing a flush at 750MB. Sorry= , I was going by the documentation. It claims that the property is around i= n 2.0.

=C2=A0

If we skip that, part of my reply still makes sense:

=C2=A0

= Having memtable_total_size_in_mb set to 20480, memtables are flushed at a r= eported value of ~2GB.

= =C2=A0

= With a constant overhead of ~10x, as suggested, this would mean that it use= d 20GB, which is 2x the size of the heap.

= =C2=A0

= That shouldn't work. According to the OS, cassandra doesn't use mor= e than ~11-12GB.

=C2=A0

From: Benedict Elliott Smith <belliottsmith@datastax.com<= /a>>
Sent: Wednesday, June 4, 2014 2:07 PM
To: u= ser@cassandra.apache.org
Subject: Re: memtable mem usage off by 10?
=C2=A0
I'm confused: there is no flush_largest_memtables_at p= roperty in C* 2.0?


On 4 June 2014 12:55, Idr=C3=A9n, Johan <Joha= n.Idren@dice.se> wrote:

Ok, so the overhead is a constant modifier, right.

=C2=A0

The 3x I arrived at with the following assumptions:

=C2=A0

hea= p is 10GB

= Default memory for memtable usage is 1/4 of heap in c* 2.0

max memory used for memtables is 2,5GB = (10/4)

flush_largest_memtables_at is 0.75

flush largest memtables when memtables use 7,5GB (3/4 of heap, 3x of the= default)

=C2=A0

With an overhead of 10x, it makes sense that my memtable is flushed when= the jmx data says it is at ~250MB, ie 2,5GB, ie 1/4 of the heap

=C2=A0

After I've set the memtable_total_size_in_mb to a value larger than = 7,5GB, it should still not go over 7,5GB on account of flush_largest_memtab= les_at, 3/4 the heap

=C2=A0

So I would expect to see memtables flushed to disk after they're bei= ng reportedly at around 750MB.

=C2=A0

Having memtable_total_size_in_mb set to 20480, memtables are flushed at = a reported value of ~2GB.

=C2=A0

With a constant overhead, this would mean that it used 20GB, which is 2x= the size of the heap, instead of 3/4 of the heap as it should be if flush_= largest_memtables_at was being respected.

=C2=A0

This shouldn't be possible.

=C2=A0

From: Benedict Elliott Smith <belliottsmith@datastax.com>
Sent: Wednesday, June 4, 2014 1:19 PM

To: u= ser@cassandra.apache.org
Subject: Re: memtable mem usage off by 10?
=C2=A0
Unfortunately it looks like the heap utilisation of memtab= les was not exposed in earlier versions, because they only maintained an es= timate.
=C2=A0
The overhead scales linearly with the amount of data in your memtables= (assuming the size of each cell is approx. constant).
=C2=A0
flush_largest_memtables_at is an independent setting to memtable_total= _space_in_mb, and generally has little effect. Ordinarily sstable flushes a= re triggered by hitting the memtable_total_space_in_mb limit. I'm afrai= d I don't follow where your 3x comes from?


On 4 June 2014 12:04, Idr=C3=A9n, Johan <Joha= n.Idren@dice.se> wrote:

Aha, ok. Thanks.

=C2=A0

Trying to understand what my cluster is doing:

=C2=A0

cassandra.db.memtable_data_size only gets me the actual data but not the memta= ble heap memory usage. Is there a way to check for heap memory usage?


I would expect to hit the flush_largest_m= emtables_at value, and this would be what causes the memtable flush to ssta= ble then? By default 0.75?


Then I would expect the amount of memory = to be used to be maximum ~3x of what I was seeing when I hadn't set mem= table_total_space_in_mb (1/4 by default, max 3/4 before a flush), instead o= f close to 10x (250mb vs 2gb).


This is of course assuming that the overhead scales linearly with the amoun= t of data in my table, we're using one table with three cells in this c= ase. If it hardly increases at all, then I'll give up I guess :)=

At least until 2.1.0 comes out and I can = compare.


BR

Johan

=C2=A0

From: Benedict Elliott Smith <belliottsmith@datastax.com>
Sent: Wednesday, June 4, 2014 12:33 PM

To: u= ser@cassandra.apache.org
Subject: Re: memtable mem usage off by 10?
=C2=A0
These measurements tell you the amount of user data stored= in the memtables, not the amount of heap used to store it, so the same app= lies.


On 4 June 2014 11:04, Idr=C3=A9n, Johan <Joha= n.Idren@dice.se> wrote:

I'm not measuring memtable size by looking at the sstables on disk, = no. I'm looking through the JMX data. So I would believe (or hope) that= I'm getting relevant data.

=C2=A0

If I have a heap of 10GB and set the memtable usage to 20GB, I would exp= ect to hit other problems, but I'm not seeing memory usage over 10GB fo= r the heap, and the machine (which has ~30gb of memory) is showing ~10GB fr= ee, with ~12GB used by cassandra, the rest in caches.

=C2=A0

Reading 8k rows/s, writing 2k rows/s on a 3 node cluster. So it's no= t idling.

=C2=A0

BR

Johan

=C2=A0

From: Benedict Elliott Smith <belliottsmith@datastax.com<= /a>>
Sent: Wednesday, June 4, 2014 11:56 AM
To: u= ser@cassandra.apache.org
Subject: Re: memtable mem usage off by 10?
=C2=A0
If you are storing small values in your columns, the objec= t overhead is very substantial. So what is 400Mb on disk may well be 4Gb in= memtables, so if you are measuring the memtable size by the resulting ssta= ble size, you are not getting an accurate picture. This overhead has been reduced by about 90% in the upcoming 2.1 r= elease, through tickets 6271, 6689 and 6694.


On 4 June 2014 10:49, Idr=C3=A9n, Johan <Joha= n.Idren@dice.se> wrote:

Hi,

=C2=A0

I'm seeing some strange behavior of the memtables, both in 1.2.13 an= d 2.0.7, basically it looks like it's using 10x less memory than it sho= uld based on the documentation and options.

=C2=A0

10GB heap for both clusters.

1.2.x should use 1/3 of the heap for memtables, but it uses max ~300mb b= efore flushing

=C2=A0

2.0.7, same but 1/4 and ~250mb

=C2=A0

In the 2.0.7 cluster I set the memtable_total_space_in_mb to 4096, which= then allowed cassandra to use up to ~400mb for memtables...

=C2=A0

I'm now running with 20480 for memtable_total_space_in_mb and cassan= dra is using ~2GB for memtables.

=C2=A0

Soo, off by 10 somewhere? Has anyone else seen this? Can't find a JI= RA for any bug connected to this.

java 1.7.0_55, JNA 4.1.0 (for the 2.0 cluster)

=C2=A0

BR

Johan

=C2=A0
=C2=A0
=C2=A0
=C2=A0

--089e0149cef61b72e804fb04243d--