Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0E1EEC2E5 for ; Fri, 28 Jun 2013 06:15:33 +0000 (UTC) Received: (qmail 95380 invoked by uid 500); 28 Jun 2013 06:15:30 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 94908 invoked by uid 500); 28 Jun 2013 06:15:15 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 94894 invoked by uid 99); 28 Jun 2013 06:15:11 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Jun 2013 06:15:11 +0000 X-ASF-Spam-Status: No, hits=1.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of sulong1984@gmail.com designates 209.85.216.67 as permitted sender) Received: from [209.85.216.67] (HELO mail-qa0-f67.google.com) (209.85.216.67) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Jun 2013 06:15:07 +0000 Received: by mail-qa0-f67.google.com with SMTP id hu16so207869qab.6 for ; Thu, 27 Jun 2013 23:14:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=2KUHpPXrGo65E0C1YkwxT/58w/8CT/IcqvRUUInBNFw=; b=xTqG50Cy86igyHEW4JK766JopvTCUHYe+u9KNQJlDLdplgPIUfTGA8jzQNQ64qWlNV rGhlqj7ZfHJOjDoJnSSdyAAVmYX484vbvmWxOMQqVwaA0wVwJ3V0hl6pz4m4xnhFSY4h 7Qez1pQ6BhR4OOQJopNMKMnsu0djtnJhyTVh9n6JrqqT9IAuT4siFBbNqEXp9mAZd+T6 UgmynO/tlPp7ONq2RonHjSVTBxa34m+LzSZhCYq5tfZnuPm5JPgQzgKedU/1Y59URvsO xVZCUDw0CDNSmpobFH9beiRslNHWcEoLi55pt9ICk2az/kiN0jP30ZFp0eC506teHKu0 fMIw== X-Received: by 10.224.88.74 with SMTP id z10mr16086072qal.26.1372400086404; Thu, 27 Jun 2013 23:14:46 -0700 (PDT) MIME-Version: 1.0 Received: by 10.49.94.138 with HTTP; Thu, 27 Jun 2013 23:14:25 -0700 (PDT) In-Reply-To: <5EBD1E75-9C29-44EB-9950-38064A3F1243@thelastpickle.com> References: <5EBD1E75-9C29-44EB-9950-38064A3F1243@thelastpickle.com> From: sulong Date: Fri, 28 Jun 2013 14:14:25 +0800 Message-ID: Subject: Re: CompactionExecutor holds 8000+ SSTableReader 6G+ memory To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=001a1132e00aa3d2d104e030cb1b X-Virus-Checked: Checked by ClamAV on apache.org --001a1132e00aa3d2d104e030cb1b Content-Type: text/plain; charset=UTF-8 aaron, thanks for your reply. Yes, I do use the Leveled compactions strategy, and the SSTable size is 10M. If it happens again, I will try to enlarge the sstable size. I just wonder why cassandra doesn't limit the SSTableReader's total memory usage when compacting. Lots of memory are consumed by the SSTableReader's cache. Why not clear these cache first at the beginning of compaction? On Fri, Jun 28, 2013 at 1:14 PM, aaron morton wrote: > Are you running the Levelled compactions strategy ? > If so what is the max SSTable size and what is the total data per node? > > If you are running it try using a larger SSTable size like 32MB > > Cheers > > ----------------- > Aaron Morton > Freelance Cassandra Consultant > New Zealand > > @aaronmorton > http://www.thelastpickle.com > > On 27/06/2013, at 2:02 PM, sulong wrote: > > According to the OpsCenter records, yes, the compaction was running > then, 8.5mb /s > > > On Thu, Jun 27, 2013 at 9:54 AM, sulong wrote: > >> version: 1.2.2 >> cluster read requests 800/s, write request 22/s >> Sorrry, I don't know whether the compaction was running then. >> >> >> On Thu, Jun 27, 2013 at 1:02 AM, Robert Coli wrote: >> >>> On Tue, Jun 25, 2013 at 10:13 PM, sulong wrote: >>> > I have 4 nodes cassandra cluster. Every node has 32G memory, and the >>> > cassandra jvm uses 8G. The cluster is suffering from gc. Looks like >>> > CompactionExecutor thread holds too many SSTableReader. See the >>> attachement. >>> >>> What version of Cassandra? >>> What workload? >>> Is compaction actually running? >>> >>> =Rob >>> >> >> > > --001a1132e00aa3d2d104e030cb1b Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
aaron, thanks for your reply. Yes, I do use the Leveled co= mpactions strategy, and the SSTable size is 10M. If it happens again, I wil= l try to enlarge the sstable size.=C2=A0

I just wonder w= hy cassandra doesn't limit the SSTableReader's total memory usage w= hen compacting. Lots of memory are consumed by the SSTableReader's cach= e. Why not clear these cache first at the beginning of compaction?


On Fri,= Jun 28, 2013 at 1:14 PM, aaron morton <aaron@thelastpickle.com&= gt; wrote:
Are you = running the Levelled compactions strategy ?=C2=A0
If so what is the max= SSTable size and what is the total data per node?

=C2=A0If you are running it try using a larger SSTable = size like 32MB

Cheers
=C2=A0
<= div>
-----------------
Aaron Morton
Freelance Cassandra= Consultant
New Zealand


On 27/06/2013, at 2:02 PM, sulong <sulong1984@gmail.com> wrote:
According to =C2=A0the OpsC= enter records, yes, =C2=A0the compaction was running then, 8.5mb /s


On Thu, Jun 2= 7, 2013 at 9:54 AM, sulong <sulong1984@gmail.com> wrote:<= br>
version: 1.2.2
cluster = read requests 800/s, write request 22/s
Sorrry, I don't know = whether =C2=A0the compaction was running then.


On Thu, Jun 27, 2013 at 1:02 AM, Robert Coli <rcoli@eventbrite.com= > wrote:
On Tue, Jun 25, 2013 at 10:13 PM, sulong <sulong1984@gmail.com> wrote:
> I have 4 nodes cassandra cluster. Every node has 32G memory, and the > cassandra jvm uses 8G. The cluster is suffering from gc. Looks like > CompactionExecutor thread holds too many SSTableReader. See the attach= ement.

What version of Cassandra?
What workload?
Is compaction actually running?

=3DRob




--001a1132e00aa3d2d104e030cb1b--