From user-return-64822-archive-asf-public=cust-asf.ponee.io@cassandra.apache.org Wed Dec 4 13:04:38 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id B66F4180656 for ; Wed, 4 Dec 2019 14:04:37 +0100 (CET) Received: (qmail 64506 invoked by uid 500); 4 Dec 2019 13:04:34 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 64473 invoked by uid 99); 4 Dec 2019 13:04:34 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Dec 2019 13:04:34 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id AB60A180011 for ; Wed, 4 Dec 2019 13:04:33 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.35 X-Spam-Level: X-Spam-Status: No, score=0.35 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=0.2, HTTPS_HTTP_MISMATCH=0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-ec2-va.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id cGxJNfjnErRy for ; Wed, 4 Dec 2019 13:04:29 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=209.85.222.182; helo=mail-qk1-f182.google.com; envelope-from=shishirroy2000@gmail.com; receiver= Received: from mail-qk1-f182.google.com (mail-qk1-f182.google.com [209.85.222.182]) by mx1-ec2-va.apache.org (ASF Mail Server at mx1-ec2-va.apache.org) with ESMTPS id C6F37BC530 for ; Wed, 4 Dec 2019 13:04:28 +0000 (UTC) Received: by mail-qk1-f182.google.com with SMTP id q28so6994608qkn.10 for ; Wed, 04 Dec 2019 05:04:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=a5HRbGUPBKIN3sbOKI10ru07EHBBJ6y0G7VA/2bQcI8=; b=Q5Zjb6QrgNIEilUr2h1o7/LtJH0lcN6Lxie1GWp/mpJYeKFZ2kv025yVpPIVXUjxRM K6Gq9k8TqgADXfI5A9eIgmQknahkLzy1i8Eaicye+9yUQf2L63fT+Et8jpDkUS0DkHqg S9z1P8DaynYyiFBZtE40h5SPOhDfZirPsHcl62yOVJZKkV1Rq1sGWgBZ4OdbE95piXra PzmxeK+UR5mWD3DtfKp3Z299OZvwACvqxEGFxNifA1eKdMQvSR49oeCq6HytxKxmTPN8 zflHg34asRolk1WzBPtBNxjDtjS8dQFqQpRdHV3MdrW58JP9F78P4Vbjw0h68BugsXQL LrxA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=a5HRbGUPBKIN3sbOKI10ru07EHBBJ6y0G7VA/2bQcI8=; b=A4D8B6TuM6gFEqpsGsFzA3PXyk8drXqyoJhHXWMgS/HcdgH1KsEEXgYfjbVFH15d0P E1RFD/4BbN4rHijq5wtPNLE2oOgVbxndzAO1GlbhIvi+tXP2T9GSSvgwK+hN52DYbVm7 osmRpiwS6dfHbqO5l/2B7uLYTPODcqwD8UGACvB/GNa416tj+MOywVl+AHVUD6JPMrl+ OVkKOv96UhfehTn2FYNm9pCmvJXTKKldlLUBlSvjGBB/pEB6kHLY7Apx3Yq+itjCzfxe nzQe/4OFUPljygstfxmp5iK+uAOGzEdXnwK5rZvtD2sfLMBU8FR0F6vBhRpclsGSn3VR YTVw== X-Gm-Message-State: APjAAAXLX809rkkoMm5xZAy4EAMRyIsRRX1gTmO6GRvBRats562hEup4 0PLYH4016NEcTxVbsEMI1ihI5NKkO0sklroC5RIY8wRU X-Google-Smtp-Source: APXvYqye1KD3WPALi10r7QdptFgd32WsE5AEjGRpC0ZxuIBZEQFdL7yIxsFNLJtsaE32Gm7blJN4+is1t8TmzxJYJog= X-Received: by 2002:a37:9f94:: with SMTP id i142mr2813185qke.244.1575464668046; Wed, 04 Dec 2019 05:04:28 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Shishir Kumar Date: Wed, 4 Dec 2019 18:34:16 +0530 Message-ID: Subject: Re: "Maximum memory usage reached (512.000MiB), cannot allocate chunk of 1.000MiB" To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary="000000000000e52ce20598e07370" --000000000000e52ce20598e07370 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Correct. Normally one should avoid this, as performance might degrade, but system will not die (until process gets paged out). In production we haven't done this (just changed mmap_index_only). We have an environment which gets used for customer to train/beta test that grows rapidly. Investing on infra do not make sense from cost prospective, so swap as option. But here if environment is up running it will be interesting to understand what is consuming memory and is infra sized correctly. -Shishir On Wed, 4 Dec 2019, 16:13 Hossein Ghiyasi Mehr, wrote: > "3. Though Datastax do not recommended and recommends Horizontal scale, s= o > based on your requirement alternate old fashion option is to add swap > space." > Hi Shishir, > swap isn't recommended by DataStax! > > *-------------------------------------------------------* > *VafaTech.com - A Total Solution for Data Gathering & Analysis* > *-------------------------------------------------------* > > > On Tue, Dec 3, 2019 at 5:53 PM Shishir Kumar > wrote: > >> Options: Assuming model and configurations are good and Data size per >> node less than 1 TB (though no such Benchmark). >> >> 1. Infra scale for memory >> 2. Try to change disk_access_mode to mmap_index_only. >> In this case you should not have any in memory DB tables. >> 3. Though Datastax do not recommended and recommends Horizontal scale, s= o >> based on your requirement alternate old fashion option is to add swap sp= ace. >> >> -Shishir >> >> On Tue, 3 Dec 2019, 15:52 John Belliveau, >> wrote: >> >>> Reid, >>> >>> I've only been working with Cassandra for 2 years, and this echoes my >>> experience as well. >>> >>> Regarding the cache use, I know every use case is different, but have >>> you experimented and found any performance benefit to increasing its si= ze? >>> >>> Thanks, >>> John Belliveau >>> >>> >>> On Mon, Dec 2, 2019, 11:07 AM Reid Pinchback >>> wrote: >>> >>>> Rahul, if my memory of this is correct, that particular logging messag= e >>>> is noisy, the cache is pretty much always used to its limit (and why n= ot, >>>> it=E2=80=99s a cache, no point in using less than you have). >>>> >>>> >>>> >>>> No matter what value you set, you=E2=80=99ll just change the =E2=80=9C= reached (=E2=80=A6.)=E2=80=9D >>>> part of it. I think what would help you more is to work with the team= (s) >>>> that have apps depending upon C* and decide what your performance SLA = is >>>> with them. If you are meeting your SLA, you don=E2=80=99t care about = noisy >>>> messages. If you aren=E2=80=99t meeting your SLA, then the noisy mess= ages become >>>> sources of ideas to look at. >>>> >>>> >>>> >>>> One thing you=E2=80=99ll find out pretty quickly. There are a lot of = knobs you >>>> can turn with C*, too many to allow for easy answers on what you shoul= d >>>> do. Figure out what your throughput and latency SLAs are, and you=E2= =80=99ll know >>>> when to stop tuning. Otherwise you=E2=80=99ll discover that it=E2=80= =99s a rabbit hole you >>>> can dive into and not come out of for weeks. >>>> >>>> >>>> >>>> >>>> >>>> *From: *Hossein Ghiyasi Mehr >>>> *Reply-To: *"user@cassandra.apache.org" >>>> *Date: *Monday, December 2, 2019 at 10:35 AM >>>> *To: *"user@cassandra.apache.org" >>>> *Subject: *Re: "Maximum memory usage reached (512.000MiB), cannot >>>> allocate chunk of 1.000MiB" >>>> >>>> >>>> >>>> *Message from External Sender* >>>> >>>> It may be helpful: >>>> https://thelastpickle.com/blog/2018/08/08/compression_performance.html >>>> >>>> >>>> It's complex. Simple explanation, cassandra keeps sstables in memory >>>> based on chunk size and sstable parts. It manage loading new sstables = to >>>> memory based on requests on different sstables correctly . You should = be >>>> worry about it (sstables loaded in memory) >>>> >>>> >>>> *VafaTech.com - A Total Solution for Data Gathering & Analysis* >>>> >>>> >>>> >>>> >>>> >>>> On Mon, Dec 2, 2019 at 6:18 PM Rahul Reddy >>>> wrote: >>>> >>>> Thanks Hossein, >>>> >>>> >>>> >>>> How does the chunks are moved out of memory (LRU?) if it want to make >>>> room for new requests to get chunks?if it has mechanism to clear chunk= s >>>> from cache what causes to cannot allocate chunk? Can you point me to a= ny >>>> documention? >>>> >>>> >>>> >>>> On Sun, Dec 1, 2019, 12:03 PM Hossein Ghiyasi Mehr < >>>> ghiyasimehr@gmail.com> wrote: >>>> >>>> Chunks are part of sstables. When there is enough space in memory to >>>> cache them, read performance will increase if application requests it >>>> again. >>>> >>>> >>>> >>>> Your real answer is application dependent. For example write heavy >>>> applications are different than read heavy or read-write heavy. Real t= ime >>>> applications are different than time series data environments and ... = . >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> On Sun, Dec 1, 2019 at 7:09 PM Rahul Reddy >>>> wrote: >>>> >>>> Hello, >>>> >>>> >>>> >>>> We are seeing memory usage reached 512 mb and cannot allocate 1MB. I >>>> see this because file_cache_size_mb by default set to 512MB. >>>> >>>> >>>> >>>> Datastax document recommends to increase the file_cache_size. >>>> >>>> >>>> >>>> We have 32G over all memory allocated 16G to Cassandra. What is the >>>> recommended value in my case. And also when does this memory gets fill= ed up >>>> frequent does nodeflush helps in avoiding this info messages? >>>> >>>> --000000000000e52ce20598e07370 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Correct. Normally one should avoid this, as performa= nce might degrade, but system will not die (until process gets paged out).<= /div>

In production we haven&#= 39;t done this (just changed mmap_index_only). We have an environment which= gets used for customer to train/beta test that grows rapidly. Investing on= infra do not make sense from cost prospective, so swap as option.

But here if environment is up ru= nning it will be interesting to understand what is consuming memory and is = infra sized correctly.

-= Shishir

On Wed, 4 Dec 2019, 16:13 Hossein Ghiyasi Mehr, <ghiyasimehr@gmail.com> wrote:
"3. Though Datastax do not rec= ommended and recommends Horizontal scale, so based on your requirement alte= rnate old fashion option is to add swap space."
Hi Shishir,
= swap isn't recommended by D= ataStax!

----------------------------------------------------= ---
= VafaTech.com - A Total Solution for Data Gathering & Analysis
---------= ----------------------------------------------
=


On Tue, Dec 3, 2019 at 5:53 PM Shishir Kumar <shis= hirroy2000@gmail.com> wrote:
Options: Assuming model and configura= tions are good and Data size per node less than 1 TB (though no such Benchm= ark).

1. Infra scale for memory=C2= =A0
2. Try to change disk_access_mode to=C2=A0mmap_index_only.
In this case you should not hav= e any in memory DB tables.
3. Though Datastax do not recommended and recommend= s Horizontal scale, so based on your requirement alternate old fashion opti= on is to add swap space.

-Shishir

On Tue, 3 Dec 2019= , 15:52 John Belliveau, <belliveau.john@gmail.com> wrote:
Reid,

I've only been= working with Cassandra for 2 years, and this echoes my experience as well.=

Regarding the cache use= , I know every use case is different, but have you experimented and found a= ny performance benefit to increasing its size?=C2=A0

Thanks,=C2=A0
John Bell= iveau=C2=A0


On Mon, Dec 2, 2019, 11:07 AM Reid Pinchback <rpinchback@tripadvisor.com> wrote:

Rahul, if my memory of this is correct, that particu= lar logging message is noisy, the cache is pretty much always used to its l= imit (and why not, it=E2=80=99s a cache, no point in using less than you ha= ve).=C2=A0

=C2=A0

No matter what value you set, you=E2=80=99ll just ch= ange the =E2=80=9Creached (=E2=80=A6.)=E2=80=9D part of it.=C2=A0 I think w= hat would help you more is to work with the team(s) that have apps dependin= g upon C* and decide what your performance SLA is with them.=C2=A0 If you a= re meeting your SLA, you don=E2=80=99t care about noisy messages.=C2=A0 If yo= u aren=E2=80=99t meeting your SLA, then the noisy messages become sources o= f ideas to look at.

=C2=A0

One thing you=E2=80=99ll find out pretty quickly.=C2= =A0 There are a lot of knobs you can turn with C*, too many to allow for ea= sy answers on what you should do.=C2=A0 Figure out what your throughput and= latency SLAs are, and you=E2=80=99ll know when to stop tuning.=C2=A0 Otherwise you=E2=80=99ll discover that it=E2=80=99s a rabbit hole you can = dive into and not come out of for weeks.

=C2=A0

=C2=A0

From: = Hossein Ghiyasi Mehr = <ghiyasimehr@gmail.com>
Reply-To: "user@cassandra.apache.o= rg" <user@cassandra.apache.org= >
Date: Monday, December 2, 2019 at 10:35 AM
To: "user@cassandra.apache.org= " <user@cassandra.apache.org> Subject: Re: "Maximum memory usage reached (512.000MiB), cannot= allocate chunk of 1.000MiB"

=C2=A0

Message from External Sender

It's complex. Simple explanation, cassandra keep= s sstables in memory based on chunk size and sstable parts. It manage loadi= ng new sstables to memory based on requests on different sstables correctly= . You should be worry about it (sstables loaded in memory)


VafaTech.com -= A Total Solution for Data Gathering & Analysis

=C2=A0

=C2=A0

On Mon, Dec 2, 2019 at 6:18 PM Rahul Reddy <rahulreddy1234@gmail.com> wrote:

Thanks Hossein,

=C2=A0

How does the chunks are moved out of memory (LRU?) i= f it want to make room for new requests to get chunks?if it has mechanism t= o clear chunks from cache what causes to cannot allocate chunk? Can you poi= nt me to any documention?

=C2=A0

On Sun, Dec 1, 2019, 12:03 PM Hossein Ghiyasi Mehr &= lt;ghiyasimehr@gmail.com> wrote:<= /u>

Chunks are part of sstables. When there is enough sp= ace=C2=A0in memory to cache them, read performance will increase if applica= tion requests it again.

=C2=A0

Your real answer is application dependent. For examp= le write heavy applications are different than read heavy or read-write hea= vy. Real time applications are different than time series data environments= and ... .

=C2=A0

=C2=A0

=C2=A0

On Sun, Dec 1, 2019 at 7:09 PM Rahul Reddy <rahulreddy1234@gmail.com> wrote:

Hello,

=C2=A0

We are seeing memory usage reached 512 mb and cannot= allocate 1MB.=C2=A0 I see this because file_cache_size_mb by default set t= o 512MB.

=C2=A0

Datastax document recommends to increase the file_ca= che_size.

=C2=A0

We have 32G over all memory allocated 16G to Cassand= ra. What is the recommended value in my case. And also when does this memor= y gets filled up frequent does nodeflush helps in avoiding this info messag= es?

--000000000000e52ce20598e07370--