From user-return-3957-apmail-cassandra-user-archive=cassandra.apache.org@cassandra.apache.org Tue Apr 06 07:19:18 2010 Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 22868 invoked from network); 6 Apr 2010 07:19:18 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 6 Apr 2010 07:19:18 -0000 Received: (qmail 82013 invoked by uid 500); 6 Apr 2010 07:19:17 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 81991 invoked by uid 500); 6 Apr 2010 07:19:17 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 81982 invoked by uid 99); 6 Apr 2010 07:19:17 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 07:19:17 +0000 X-ASF-Spam-Status: No, hits=-0.2 required=10.0 tests=AWL,FREEMAIL_FROM,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of ivmaykov@gmail.com designates 209.85.160.172 as permitted sender) Received: from [209.85.160.172] (HELO mail-gy0-f172.google.com) (209.85.160.172) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 07:19:12 +0000 Received: by gyh4 with SMTP id 4so2572195gyh.31 for ; Tue, 06 Apr 2010 00:18:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:received:message-id:subject:from:to:content-type :content-transfer-encoding; bh=WUgySesoHZtAeLUkwgRqhEj6F9av/bMiQT8tdxgPPYc=; b=aYUEU45qIidGyvom83EgxxC6MuQMLVrw9b4DW32KVfcdIz3QrB+zp2CliK5X5uHerh jik31Q7mlrmnRf75GqK5Q1v6iDNLP74+f6PYtKUCtj8jPOr0nrvm3t0pUnLM1y7BCR1H Uaytheb2LBgVtE0ojH5YdwizTAz4XBbQ09RG8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type:content-transfer-encoding; b=CBCeYvF63jP4UCjWnD12vhzIMbRKs4dF6qOLMo4RqbCDnCMCHCZ0Gg2Pf3KNTtxEd7 58ZNkATvCxMz84EvcakGi07nFsYEIY9ozsaJpMAZqqctMjpM17qrcE4Zfx7N+drIHYeS cMLoQbC04XyVThFmWkweZoCogbKcnsTGYuBdw= MIME-Version: 1.0 Received: by 10.151.10.10 with HTTP; Tue, 6 Apr 2010 00:18:51 -0700 (PDT) In-Reply-To: References: <4BBADBD9.6030907@digg.com> Date: Tue, 6 Apr 2010 00:18:51 -0700 Received: by 10.150.210.19 with SMTP id i19mr7514266ybg.105.1270538331759; Tue, 06 Apr 2010 00:18:51 -0700 (PDT) Message-ID: Subject: Re: Overwhelming a cluster with writes? From: Ilya Maykov To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Right, I meant 4GB heap vs. the standard 1GB. And all other options in cassandra.in.sh at their defaults. Sorry I am a bit new to JVM tuning, and very new to Cassandra :) -- Ilya On Tue, Apr 6, 2010 at 12:16 AM, Benjamin Black wrote: > I am specifically suggesting you NOT use a heap that large with your > 8GB machines. =A0Please test with 4GB first. > > On Tue, Apr 6, 2010 at 12:13 AM, Ilya Maykov wrote: >> That does sound similar. It's possible that the difference I'm seeing >> between ConsistencyLevel.ZERO and ConsistencyLevel.ALL is simply due >> to the fact that using ALL slows down the writers enough that the GC >> can keep up. I could do a test with multiple clients writing at ALL in >> parallel tomorrow. If there are still no problems writing at ALL even >> with extra load from additional clients, that might point to problems >> in how async writes are handled vs. sync writes. >> >> I will also do some profiling of the server processes with both ZERO >> and ALL writer behaviors and report back. >> >> RE: JVM_OPTS, I will try running with the "more sane" options (but a >> larger heap) as well. >> >> -- Ilya >> >> On Mon, Apr 5, 2010 at 11:59 PM, Rob Coli wrote: >>> On 4/5/10 11:48 PM, Ilya Maykov wrote: >>>> >>>> No, the disks on all nodes have about 750GB free space. Also as >>>> mentioned in my follow-up email, writing with ConsistencyLevel.ALL >>>> makes the slowdowns / crashes go away. >>> >>> I am not sure if the above is consistent with the cause of #896, but th= e >>> other symptoms ("I inserted a bunch of data really fast via Thrift and = GC >>> melted my machine!") sound like it.. >>> >>> https://issues.apache.org/jira/browse/CASSANDRA-896 >>> >>> =3DRob >>> >> >