Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 91279 invoked from network); 2 Apr 2011 05:45:54 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 2 Apr 2011 05:45:54 -0000 Received: (qmail 73217 invoked by uid 500); 2 Apr 2011 05:45:52 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 72540 invoked by uid 500); 2 Apr 2011 05:45:52 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 72532 invoked by uid 99); 2 Apr 2011 05:45:51 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Apr 2011 05:45:51 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,X_IP X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of alienth@gmail.com designates 209.85.210.60 as permitted sender) Received: from [209.85.210.60] (HELO mail-pz0-f60.google.com) (209.85.210.60) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Apr 2011 05:45:45 +0000 Received: by pzk9 with SMTP id 9so631589pzk.25 for ; Fri, 01 Apr 2011 22:45:24 -0700 (PDT) MIME-Version: 1.0 Received: by 10.142.210.16 with SMTP id i16mr595671wfg.47.1301723124741; Fri, 01 Apr 2011 22:45:24 -0700 (PDT) Received: by t19g2000prd.googlegroups.com with HTTP; Fri, 1 Apr 2011 22:45:24 -0700 (PDT) Date: Fri, 1 Apr 2011 22:45:24 -0700 (PDT) In-Reply-To: X-IP: 98.210.57.101 References: <2a7b7433-adc1-44fc-aeb2-ead4ea5257d9@w9g2000prg.googlegroups.com> User-Agent: G2/1.0 X-HTTP-UserAgent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110221 Ubuntu/10.10 (maverick) Firefox/3.6.14,gzip(gfe) Message-ID: <2c627de3-2238-4cce-99e3-3502fc1b9bb8@t19g2000prd.googlegroups.com> Subject: Re: Bizarre side-effect of increasing read concurrency From: Jason Harvey To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Ah, that would probably explain it. Thanks! On Apr 1, 8:49=A0pm, Edward Capriolo wrote: > On Fri, Apr 1, 2011 at 11:27 PM, Jason Harvey wrote: > > On further analysis, it looks like this behavior occurs when a node is > > simply restarted. Is that normal behavior? If mark-and-sweep becomes > > less and less effective over time, does that suggest an issue with GC, > > or an issue with memory use? > > > On Apr 1, 8:21=A0pm, Jason Harvey wrote: > >> After increasing read concurrency from 8 to 64, GC mark-and-sweep was > >> suddenly able to reclaim much more memory than it previously did. > > >> Previously, mark-and-sweep would run around 5.5GB, and would cut heap > >> usage to 4GB. Now, it still runs at 5.5GB, but it shrinks all the way > >> down to 2GB used. This behavior was consistent in every machine I > >> increased read-concurrent on. > > >> Any thoughts on why this behavior changed? No other diagnostics > >> appeared to correlate to the concurrency change, besides thread count. > > Jason, > > First you do not need to restart to adjust concurrent readers. It can > be done from JMX without restart. > > As for the memory, after you restart you may have drained your caches > and memtables which explains why less memory is used. > > Java also enjoys using all the memory your allocate and the Garbage > collection does not give it back unless it needs to. > > Edward