Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 48537 invoked from network); 2 Apr 2011 03:27:49 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 2 Apr 2011 03:27:49 -0000 Received: (qmail 13123 invoked by uid 500); 2 Apr 2011 03:27:47 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 13100 invoked by uid 500); 2 Apr 2011 03:27:47 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 13092 invoked by uid 99); 2 Apr 2011 03:27:46 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Apr 2011 03:27:46 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=FREEMAIL_FROM,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL,X_IP X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of alienth@gmail.com designates 209.85.160.188 as permitted sender) Received: from [209.85.160.188] (HELO mail-gy0-f188.google.com) (209.85.160.188) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 02 Apr 2011 03:27:39 +0000 Received: by gyg4 with SMTP id 4so7978332gyg.25 for ; Fri, 01 Apr 2011 20:27:18 -0700 (PDT) MIME-Version: 1.0 Received: by 10.91.160.32 with SMTP id m32mr22436ago.14.1301714838230; Fri, 01 Apr 2011 20:27:18 -0700 (PDT) Received: by w9g2000prg.googlegroups.com with HTTP; Fri, 1 Apr 2011 20:27:18 -0700 (PDT) Date: Fri, 1 Apr 2011 20:27:18 -0700 (PDT) In-Reply-To: X-IP: 98.210.57.101 References: User-Agent: G2/1.0 X-HTTP-UserAgent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.14) Gecko/20110221 Ubuntu/10.10 (maverick) Firefox/3.6.14,gzip(gfe) Message-ID: <2a7b7433-adc1-44fc-aeb2-ead4ea5257d9@w9g2000prg.googlegroups.com> Subject: Re: Bizarre side-effect of increasing read concurrency From: Jason Harvey To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org On further analysis, it looks like this behavior occurs when a node is simply restarted. Is that normal behavior? If mark-and-sweep becomes less and less effective over time, does that suggest an issue with GC, or an issue with memory use? On Apr 1, 8:21=A0pm, Jason Harvey wrote: > After increasing read concurrency from 8 to 64, GC mark-and-sweep was > suddenly able to reclaim much more memory than it previously did. > > Previously, mark-and-sweep would run around 5.5GB, and would cut heap > usage to 4GB. Now, it still runs at 5.5GB, but it shrinks all the way > down to 2GB used. This behavior was consistent in every machine I > increased read-concurrent on. > > Any thoughts on why this behavior changed? No other diagnostics > appeared to correlate to the concurrency change, besides thread count.