Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C05B1200C7D for ; Tue, 2 May 2017 04:36:53 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id BF01F160BC2; Tue, 2 May 2017 02:36:53 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id DDED6160BC1 for ; Tue, 2 May 2017 04:36:52 +0200 (CEST) Received: (qmail 48715 invoked by uid 500); 2 May 2017 02:36:51 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 48705 invoked by uid 99); 2 May 2017 02:36:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 May 2017 02:36:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 9309A1AFC3E for ; Tue, 2 May 2017 02:36:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.93 X-Spam-Level: * X-Spam-Status: No, score=1.93 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RCVD_IN_SORBS_SPAM=0.5, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd2-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id RgI2PnJD0MCz for ; Tue, 2 May 2017 02:36:48 +0000 (UTC) Received: from mail-qt0-f182.google.com (mail-qt0-f182.google.com [209.85.216.182]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id E24BF5F3BC for ; Tue, 2 May 2017 02:36:47 +0000 (UTC) Received: by mail-qt0-f182.google.com with SMTP id m36so101877320qtb.0 for ; Mon, 01 May 2017 19:36:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=/ZkuE6FxaUhjX593Ce4BX5wItT2V8LlXz3BK8eEtXlQ=; b=KXgz39YIXUx5iECEFbPoPjBQ9MEjfkf+olziqkycJvVtGSqyiRGF0uDkBxoHDXXF/B D5OxB8M2n/XrTrepDcloVHQHsSEJnltH7+hPNfM+sss5muCAaRWyvXB9Nx0rqtatO2nz q83BUtFGNTVNxet3DYzdM6Fg5cQ2wfklUPLEWY3aWaDBsftHXZk61+M/cIGXJoNFPUkB FMUZ1lZPXJR3bdWloK7rJQq1lj/Yoh5yhZOVKjZi+447es1rm5q4elFqWhjmoMkxs9eW sMC7hflGcpMCbRg49ZCaYopRAZ12PgtBX/Ycjjl/iMPNf8Ch7CfSnyCs5CQQPKDX46sf akrQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=/ZkuE6FxaUhjX593Ce4BX5wItT2V8LlXz3BK8eEtXlQ=; b=EUy203XLxYH/Fc1cJXQavNtxi2fH4fM2HfGZD6s3LCR6zSDumM2chkVOP1PMB4DfL3 f8Ls2QrpBi9bonY/9isvTgDd9dOKNXxX/x5xPdEyyeik1nmcy/vfmefNSoF0epzEcrzv UhBIw5aXirKrpphzyRgXv0ISgFz1JB6FSM9k/M7XxsVVv2jD5DTBC6VyS3WKvSFSr0kk ZLY6arr/o3JBRwZLAxIQvaEAZS7S+LJtwToUyydhYeSs2ohB8plyuP+9jo+Uz7pFwijk lqD/szMrbGYgAW9mP6ePEXeZIdt44CbeJVmGMqS4IpMTRk5jrMTS2guUChOS8MtRuUQu wsCA== X-Gm-Message-State: AN3rC/52CziIzi4nPC54+Y+JLseJ7DGntkW0FatEp70nXsiggfi4lXND jjq+I7ZXsIBccAzAJ7op4QsRiT4weQ== X-Received: by 10.237.38.102 with SMTP id z93mr27022692qtc.292.1493692606962; Mon, 01 May 2017 19:36:46 -0700 (PDT) MIME-Version: 1.0 Received: by 10.140.101.97 with HTTP; Mon, 1 May 2017 19:36:46 -0700 (PDT) In-Reply-To: References: From: Chris Lohfink Date: Mon, 1 May 2017 21:36:46 -0500 Message-ID: Subject: Re: what is MemtableReclaimMemory mean ?? To: Pranay akula Cc: Alain RODRIGUEZ , user@cassandra.apache.org Content-Type: multipart/alternative; boundary=94eb2c1229483de07b054e816950 archived-at: Tue, 02 May 2017 02:36:53 -0000 --94eb2c1229483de07b054e816950 Content-Type: text/plain; charset=UTF-8 Question though, how many tables do you have? If you have more than a few hundreds it could be bottlenecking the flushing if it is flushing very frequently. On Mon, May 1, 2017 at 9:32 PM, Chris Lohfink wrote: > Theres a read barrier to stop reclaiming a memtable when there are > requests actively reading it. The *MemtableReclaimMemory* pool offloads > that wait instead of blocking the caller. It in itself is not going to use > any cpu or increase load. It will however block the releasing of the > memtable resources which might cause additional heap allocation pressure. > Its more likely a symptom of GCs or reads being slow than the cause of the > issue however. > > Chris > > On Mon, May 1, 2017 at 9:01 PM, Pranay akula > wrote: > >> Hi Alain, >> >> when "*MemtableReclaimMemory*" Pending Tasks increasing, its slowly >> backing up reads and writes mostly writes. yes i am seeing bit high GC >> pressure, currently we are using 24Gb Heap and G1GC collection. I tried >> changing Memtable flush threshold it did helped a little but not much. I am >> not seeing any Errors in the Logs. >> >> >> Thanks >> Pranay. >> >> On Thu, Apr 27, 2017 at 6:08 AM, Alain RODRIGUEZ >> wrote: >> >>> Hi Pranay, >>> >>> According to http://docs.datastax.com/en/ca >>> ssandra/3.0/cassandra/tools/toolsTPstats.html, "*MemtableReclaimMemory*" >>> is the thread pool used for "Making unused memory available". I don't know >>> much about it since it was never an issue for me. Neither did I heard much >>> about it. >>> >>> >>> - Are pending tasks staying high for a long period? `watch -d >>> nodetool tpstats` >>> - What are your GC settings? >>> - Any other threads pending, blocked or dropped? >>> - Do you have errors or warnings in your logs? >>> - Any GC pressure? (monitored through charts or logs at INFO level, >>> or WARN on recent versions) >>> >>> >>> C*heers, >>> ----------------------- >>> Alain Rodriguez - @arodream - alain@thelastpickle.com >>> France >>> >>> The Last Pickle - Apache Cassandra Consulting >>> http://www.thelastpickle.com >>> >>> >>> >>> 2017-04-16 16:04 GMT+02:00 Pranay akula : >>> >>>> Hi, >>>> >>>> what is *MemtableReclaimMemory* mean in nodetooltpstats ?? does this >>>> mean trying to flushing memtable from memory to SStables. >>>> >>>> I can see sometimes increase in pending tasks of MemtableReclaimMemory >>>> in nodetool tpstats, at that time i can see increase in load on those nodes. >>>> >>>> Does decreasing memtable_cleanup_threshold will help ?? >>>> >>>> Thanks >>>> Pranay. >>>> >>> >>> >> > --94eb2c1229483de07b054e816950 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Question though, how many tables do you have? If you have = more than a few hundreds it could be bottlenecking the flushing if it is fl= ushing very frequently.

On Mon, May 1, 2017 at 9:32 PM, Chris Lohfink <clohfink85@g= mail.com> wrote:
Theres a read barrier to stop reclaiming a memtable when there are= requests actively reading it. The=C2=A0Memta= bleReclaimMemory=C2=A0pool offloads that wait instead of blocking the c= aller. It in itself is not going to use any cpu or increase load. It will h= owever block the releasing of the memtable resources which might cause addi= tional heap allocation pressure. Its more likely a symptom of GCs or reads = being slow than the cause of the issue however.

Chris

On Mon, May 1, 2017 at 9:01 PM, Pranay akula <p= ranay.akula2323@gmail.com> wrote:
Hi Alain,

when =C2=A0"MemtableRe= claimMemory" =C2=A0Pending Tasks = increasing, its slowly backing up reads and writes mostly writes. yes i am = seeing bit high GC pressure, currently we are using 24Gb Heap =C2=A0and G1G= C collection. I tried changing Memtable flush threshold it did helped a lit= tle but not much. I am not seeing any Errors in the Logs.
=

Thanks<= /span>
Pranay.
On Thu, Apr 27, 2017 at 6:08 AM, Alain RODRIGU= EZ <arodrime@gmail.com> wrote:
Hi Pranay,

According to http://docs.datastax.com/en/cassandra/3.0/cass= andra/tools/toolsTPstats.html, "MemtableReclaimMemory" is the th= read pool used for "Making unused memory available". I don't = know much about it since it was never an issue for me. Neither did I heard = much about it.

  • Are pending tasks staying high for a long period? `= watch -d nodetool tpstats`
  • What are your GC settings?
  • Any o= ther threads pending, blocked or dropped?
  • Do you have errors or war= nings in your logs?
  • Any GC pressure? (monitored through charts or l= ogs at INFO level, or WARN on recent versions)

C*heers,
-----------------------
Alain Rodr= iguez - @arodream - alain@thelastpickle.com
France

The Last Pickle - Apache Cassandra Consulting



2017-04-16 16:04 GMT+02:00 Pranay akula <pranay.akula2323= @gmail.com>:
Hi,

what is MemtableReclaimMemory mean in node= tooltpstats ?? does this mean trying to flushing memtable from memory to SS= tables.

I can see sometimes increase in pending task= s of =C2=A0MemtableReclaimMemory in nodetool tpstats, at that time i can se= e increase in load on those nodes.

Does decreasing= =C2=A0memtable_cleanup_threshold will help ??

Thanks
Pranay.




--94eb2c1229483de07b054e816950--