Return-Path: X-Original-To: apmail-cassandra-user-archive@www.apache.org Delivered-To: apmail-cassandra-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C0E577344 for ; Thu, 20 Oct 2011 17:58:32 +0000 (UTC) Received: (qmail 98155 invoked by uid 500); 20 Oct 2011 17:58:30 -0000 Delivered-To: apmail-cassandra-user-archive@cassandra.apache.org Received: (qmail 98073 invoked by uid 500); 20 Oct 2011 17:58:30 -0000 Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@cassandra.apache.org Delivered-To: mailing list user@cassandra.apache.org Received: (qmail 98065 invoked by uid 99); 20 Oct 2011 17:58:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Oct 2011 17:58:30 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of edlinuxguru@gmail.com designates 209.85.210.172 as permitted sender) Received: from [209.85.210.172] (HELO mail-iy0-f172.google.com) (209.85.210.172) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 20 Oct 2011 17:58:22 +0000 Received: by iabn5 with SMTP id n5so4127578iab.31 for ; Thu, 20 Oct 2011 10:58:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Fo0Rz17r/VNQ65PTlW2ncHk6T2qSd9Xnj8KXpcgafCQ=; b=I57WpDppI4uNIDESrKx1T/uZ0jOPyxzK6TnniCFviPLZDq+plX60W9eaFFoaeh5QAy EkEM8Iqy7ZdC7Y2UKFHl8VR2puyMsfYZ6gWSVi+cyBfANaauCKkp9Pa1Bf3524jITLMB xMSi92ZS3y4GXTMejgnNteRWKPCuwmTX9i/LI= MIME-Version: 1.0 Received: by 10.42.154.201 with SMTP id r9mr19821772icw.14.1319133481547; Thu, 20 Oct 2011 10:58:01 -0700 (PDT) Received: by 10.42.2.79 with HTTP; Thu, 20 Oct 2011 10:58:01 -0700 (PDT) In-Reply-To: <4ea051fc.0476ec0a.3048.2246@mx.google.com> References: <4ea051fc.0476ec0a.3048.2246@mx.google.com> Date: Thu, 20 Oct 2011 13:58:01 -0400 Message-ID: Subject: Re: Cassandra 1.0.0 - Node Load Bug From: Edward Capriolo To: user@cassandra.apache.org Content-Type: multipart/alternative; boundary=90e6ba6e82da94389f04afbeb262 X-Virus-Checked: Checked by ClamAV on apache.org --90e6ba6e82da94389f04afbeb262 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On Thu, Oct 20, 2011 at 12:53 PM, Dan Hendry wro= te: > I have been playing around with Cassandra 1.0.0 in our test environment i= t > seems pretty sweet so far. I have however come across what appears to be = a > bug tracking node load. I have enabled compression and levelled compactio= n > on all CFs (scrub + snapshot deletion) and the nodes have been operating > normally for a day or two. I started getting concerned when the load as > reported by nodetool ring kept increasing (it seems monotonically) despit= e > seeing a compression ratio of ~2.5x (as a side note, I find it strange > Cassandra does not provide the compression ratio via jmx or in the logs).= I > initially thought there might be a bug in cleaning up obsolete SSTables b= ut > I then noticed the following discrepancy: > > > > Nodetool ring reports: > > 10.112.27.65 datacenter1 rack1 Up Normal 8.= 64 > GB 50.00% 170141183460469231731687303715884105727 > > > > Yet du . =96h reports: only 2.4G in the data directory. > > > > After restarting the node, nodetool ring reports a more accurate: > > 10.112.27.65 datacenter1 rack1 Up Normal 2.35 GB > 50.00% 170141183460469231731687303715884105727 > > > > Again, both compression and levelled compaction have been enabled on all > CFs. Is this a known issue or has anybody else observed a similar pattern= ? > > > > Dan Hendry > > (403) 660-2297 > > > In 0.7.8 known bug. Scrub would 'orphan' files that way and the counts woul= d be off like this. If you restart the node and the counts like equal again i= t might be some type of regression. --90e6ba6e82da94389f04afbeb262 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: quoted-printable
On Thu, Oct 20, 2011 at 12:53 PM, Dan Hendry= <dan.hen= dry.junk@gmail.com> wrote:

I have been playing around with Cassandra 1.0.0 in our test environment= it seems pretty sweet so far. I have however come across what appears to b= e a bug tracking node load. I have enabled compression and levelled compact= ion on all CFs (scrub=A0 + snapshot deletion) and the nodes have been opera= ting normally for a day or two. I started getting concerned when the load a= s reported by nodetool ring kept increasing (it seems monotonically) despit= e seeing a compression ratio of ~2.5x (as a side note, I find it strange Ca= ssandra does not provide the compression ratio via jmx or in the logs). I i= nitially thought there might be a bug in cleaning up obsolete SSTables but = I then noticed the following discrepancy:

=A0

Nodetool ring reports:=

=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 10= .112.27.65=A0=A0=A0 datacenter1 rack1=A0=A0=A0=A0=A0=A0 Up=A0=A0=A0=A0 Norm= al=A0 8.64 GB=A0=A0=A0=A0=A0=A0=A0=A0 50.00%=A0 170141183460469231731687303= 715884105727

=A0

Yet du . =96h reports:= only 2.4G in the data directory.

=A0

After restarting the node, nodetool ring reports a more accu= rate:

10.112.27.65=A0=A0=A0 datacenter1 rack1=A0=A0=A0=A0=A0=A0 Up=A0=A0=A0=A0 No= rmal=A0 2.35 GB=A0=A0=A0=A0=A0=A0=A0=A0 50.00%=A0 1701411834604692317316873= 03715884105727

=A0

Agai= n, both compression and levelled compaction have been enabled on all CFs. I= s this a known issue or has anybody else observed a similar pattern?

=A0

Dan Hendry

(403) 660-2297

=A0



In 0.7.8 known bug. Scrub would &#= 39;orphan' files that way and the counts would be off like this. If you= restart the node and the counts like equal again it might be some type of = regression.
--90e6ba6e82da94389f04afbeb262--