Return-Path: Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: (qmail 23187 invoked from network); 6 Apr 2010 07:20:13 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 6 Apr 2010 07:20:13 -0000 Received: (qmail 84445 invoked by uid 500); 6 Apr 2010 07:20:09 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 84268 invoked by uid 500); 6 Apr 2010 07:20:09 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 84155 invoked by uid 99); 6 Apr 2010 07:20:09 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 07:20:09 +0000 X-ASF-Spam-Status: No, hits=0.6 required=10.0 tests=AWL,RCVD_IN_DNSWL_NONE,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (athena.apache.org: local policy) Received: from [209.85.221.192] (HELO mail-qy0-f192.google.com) (209.85.221.192) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 06 Apr 2010 07:20:04 +0000 Received: by qyk30 with SMTP id 30so5785884qyk.16 for ; Tue, 06 Apr 2010 00:19:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.229.233.75 with HTTP; Tue, 6 Apr 2010 00:19:43 -0700 (PDT) In-Reply-To: References: <4BBADBD9.6030907@digg.com> Date: Tue, 6 Apr 2010 00:19:43 -0700 Received: by 10.229.213.77 with SMTP id gv13mr1810092qcb.63.1270538383081; Tue, 06 Apr 2010 00:19:43 -0700 (PDT) Message-ID: Subject: Re: Overwhelming a cluster with writes? From: Benjamin Black To: user@cassandra.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable OK, cool, looking forward to your results. On Tue, Apr 6, 2010 at 12:18 AM, Ilya Maykov wrote: > Right, I meant 4GB heap vs. the standard 1GB. And all other options in > cassandra.in.sh at their defaults. > > Sorry I am a bit new to JVM tuning, and very new to Cassandra :) > > -- Ilya > > On Tue, Apr 6, 2010 at 12:16 AM, Benjamin Black wrote: >> I am specifically suggesting you NOT use a heap that large with your >> 8GB machines. =A0Please test with 4GB first. >> >> On Tue, Apr 6, 2010 at 12:13 AM, Ilya Maykov wrote: >>> That does sound similar. It's possible that the difference I'm seeing >>> between ConsistencyLevel.ZERO and ConsistencyLevel.ALL is simply due >>> to the fact that using ALL slows down the writers enough that the GC >>> can keep up. I could do a test with multiple clients writing at ALL in >>> parallel tomorrow. If there are still no problems writing at ALL even >>> with extra load from additional clients, that might point to problems >>> in how async writes are handled vs. sync writes. >>> >>> I will also do some profiling of the server processes with both ZERO >>> and ALL writer behaviors and report back. >>> >>> RE: JVM_OPTS, I will try running with the "more sane" options (but a >>> larger heap) as well. >>> >>> -- Ilya >>> >>> On Mon, Apr 5, 2010 at 11:59 PM, Rob Coli wrote: >>>> On 4/5/10 11:48 PM, Ilya Maykov wrote: >>>>> >>>>> No, the disks on all nodes have about 750GB free space. Also as >>>>> mentioned in my follow-up email, writing with ConsistencyLevel.ALL >>>>> makes the slowdowns / crashes go away. >>>> >>>> I am not sure if the above is consistent with the cause of #896, but t= he >>>> other symptoms ("I inserted a bunch of data really fast via Thrift and= GC >>>> melted my machine!") sound like it.. >>>> >>>> https://issues.apache.org/jira/browse/CASSANDRA-896 >>>> >>>> =3DRob >>>> >>> >> >