Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (athena.apache.org: domain of edlinuxguru@gmail.com
 designates 209.85.223.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPmnz8H2q7K9fisdbYyLk1wxkG8hVurVKBBp7DkWa61YYwL4gw@mail.gmail.com>
References: 
 <CAPmnz8H2q7K9fisdbYyLk1wxkG8hVurVKBBp7DkWa61YYwL4gw@mail.gmail.com>
Date: Sun, 7 Oct 2012 10:39:29 -0400
Message-ID: 
 <CAENxBwziTuLSvmFp+T62rHys3EA-UbpGK5gNZt9HYyj=LqEfMw@mail.gmail.com>
Subject: Re: MBean cassandra.db.CompactionManager TotalBytesCompacted counts
 backwards
From: Edward Capriolo <edlinuxguru@gmail.com>
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I have not looked at this JMX object in a while, however the
compaction manager can support multiple threads. Also it moves from
0-filesize each time it has to compact a set of files.

That is more useful for showing current progress rather then lifetime history.


On Fri, Oct 5, 2012 at 7:27 PM, Bryan Talbot <btalbot@aeriagames.com> wrote:
> I've recently added compaction rate (in bytes / second) to my monitors for
> cassandra and am seeing some odd values.  I wasn't expecting the values for
> TotalBytesCompacted to sometimes decrease from one reading to the next.  It
> seems that the value should be monotonically increasing while a server is
> running -- obviously it would start again at 0 when the server is restarted
> or if the counter rolls over (unlikely for a 64 bit long).
>
> Below are two samples taken 60 seconds apart: the value decreased by
> 2,954,369,012 between the two readings.
>
> reported_metric=[timestamp:1349476449, status:200,
> request:[mbean:org.apache.cassandra.db:type=CompactionManager,
> attribute:TotalBytesCompacted, type:read], value:7548675470069]
>
> previous_metric=[timestamp:1349476389, status:200,
> request:[mbean:org.apache.cassandra.db:type=CompactionManager,
> attribute:TotalBytesCompacted, type:read], value:7551629839081]
>
>
> I briefly looked at the code for CompactionManager and a few related classes
> and don't see anyplace that is performing subtraction explicitly; however,
> there are many additions of signed long values that are not validated and
> could conceivably contain a negative value thus causing the
> totalBytesCompacted to decrease.  It's interesting to note that the all of
> the differences I've seen so far are more than the overflow value of a
> signed 32 bit value.  The OS (CentOS 5.7) and sun java vm (1.6.0_29) are
> both 64 bit.  JNA is enabled.
>
> Is this expected and normal?  If so, what is the correct interpretation of
> this metric?  I'm seeing the negatives values a few times per hour when
> reading it once every 60 seconds.
>
> -Bryan
>