Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 35CBE72A7 for ; Sun, 21 Aug 2011 15:44:16 +0000 (UTC) Received: (qmail 34599 invoked by uid 500); 21 Aug 2011 15:44:14 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 34484 invoked by uid 500); 21 Aug 2011 15:44:13 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 34476 invoked by uid 99); 21 Aug 2011 15:44:13 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 15:44:13 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of meatforums@gmail.com designates 209.85.216.44 as permitted sender) Received: from [209.85.216.44] (HELO mail-qw0-f44.google.com) (209.85.216.44) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 21 Aug 2011 15:44:05 +0000 Received: by qwc23 with SMTP id 23so3314217qwc.31 for ; Sun, 21 Aug 2011 08:43:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; bh=PUa2Q8tnOADb8Uo89HRb8x41ccbcDIutilf3DEvxNAc=; b=QB1/7ccnfQoHZs3SeoPb6czwlIFXGkYALboBIf/jXD6yQtVwJIjW6EndRJKrSK7tOt I8/hyq8iscYdv4PWEVqymYPqa4blVESveK3i0k5AK0nv/Tx7wuAIc0H2fCreJ4KJEKdZ MqrlT/F8kM8eTCjM1P68S2LQ0QqIU3Ij+DZbQ= MIME-Version: 1.0 Received: by 10.229.18.77 with SMTP id v13mr779495qca.56.1313941424518; Sun, 21 Aug 2011 08:43:44 -0700 (PDT) Received: by 10.229.16.134 with HTTP; Sun, 21 Aug 2011 08:43:44 -0700 (PDT) Date: Sun, 21 Aug 2011 08:43:44 -0700 Message-ID: Subject: Commit log fills up in less than a minute From: Anand Somani To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=0015176f0bb8dd374104ab05d316 X-Virus-Checked: Checked by ClamAV on apache.org --0015176f0bb8dd374104ab05d316 Content-Type: text/plain; charset=ISO-8859-1 Hi, 7.4, 3 node cluster, RF=3 Load has not changed much, on 2 of the 3 nodes the commit log filled up in less than a minute (did not give a chance to recover). Now have been running this cluster for abt 2-3 months without any problem. At this point I do not see any unusual load (continue to investigate). The commit log has never taken more than 20% before this!!! The only changes was a re-intro of a node couple days ago with a bunch of cleanup on each node. Has anybody seen anything like this? I am starting the entire cluster again, anyway hints and things I should watch for (besides commit -log). I have backed up the commit logs and logs, so any tips on how to do a analysis on this would help. Thanks Anand --0015176f0bb8dd374104ab05d316 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi,

7.4, 3 node cluster, RF=3D3

Load has not changed much, on= 2 of the 3 nodes the commit log filled up in less than a minute (did not g= ive a chance to recover). Now have been running this cluster for abt 2-3 mo= nths without any problem. At this point I do not see any unusual load (cont= inue to investigate). The commit log has never taken more than 20% before t= his!!!

The only changes was a re-intro of a node couple days ago with a bunch = of cleanup on each node. Has anybody seen anything like this?

I am s= tarting the entire cluster again, anyway hints and things I should watch fo= r (besides commit -log). I have backed up the commit logs and logs, so any = tips on how to do a analysis on this would help.

Thanks
Anand
--0015176f0bb8dd374104ab05d316--