Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of btalbot@aeriagames.com
 designates 74.125.149.250 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAENxBwziTuLSvmFp+T62rHys3EA-UbpGK5gNZt9HYyj=LqEfMw@mail.gmail.com>
References: 
 <CAPmnz8H2q7K9fisdbYyLk1wxkG8hVurVKBBp7DkWa61YYwL4gw@mail.gmail.com>
	<CAENxBwziTuLSvmFp+T62rHys3EA-UbpGK5gNZt9HYyj=LqEfMw@mail.gmail.com>
Date: Mon, 8 Oct 2012 12:14:18 -0700
Message-ID: 
 <CAPmnz8H=ASo4-U5wEHtsxehjxANfxbMp4tRsLUf8qT0g4esjNA@mail.gmail.com>
Subject: Re: MBean cassandra.db.CompactionManager TotalBytesCompacted counts
 backwards
From: Bryan Talbot <btalbot@aeriagames.com>
To: user@cassandra.apache.org
Content-Type: multipart/alternative; boundary=047d7b6766dc37812004cb910778

--047d7b6766dc37812004cb910778
Content-Type: text/plain; charset=UTF-8

I'm attempting to plot how "busy" the node is doing compactions but there
seems to only be a few metrics reported that might be suitable:
CompletedTasks, PendingTasks, TotalBytesCompacted,
TotalCompactionsCompleted.

It's not clear to me what the difference between CompletedTasks and
TotalCompactionsCompleted is but I am plotting TotalCompactionsCompleted /
sec as one metric; however, this rate is nearly always less than 1 and
doesn't capture how much resources are used doing the compaction.  A
compaction of 4 smallest SSTables counts the same as a compaction of 4
largest SSTables but the cost is hugely different.  Thus, I'm also plotting
TotalBytesCompacted / sec.

Since the TotalBytesCompacted value sometimes moves backwards I'm not
confident that it's reporting what it is meant to report.  The code and
comments indicate that it should only be incremented by the final size of
the newly created SSTable or by the bytes-compacted-so-far for a larger
compaction, so I don't see why it should be reasonable for it to sometimes
decrease.

How should the impact of compaction be measured if not by bytes compacted?

-Bryan


On Sun, Oct 7, 2012 at 7:39 AM, Edward Capriolo <edlinuxguru@gmail.com>wrote:

> I have not looked at this JMX object in a while, however the
> compaction manager can support multiple threads. Also it moves from
> 0-filesize each time it has to compact a set of files.
>
> That is more useful for showing current progress rather then lifetime
> history.
>
>
>
> On Fri, Oct 5, 2012 at 7:27 PM, Bryan Talbot <btalbot@aeriagames.com>
> wrote:
> > I've recently added compaction rate (in bytes / second) to my monitors
> for
> > cassandra and am seeing some odd values.  I wasn't expecting the values
> for
> > TotalBytesCompacted to sometimes decrease from one reading to the next.
>  It
> > seems that the value should be monotonically increasing while a server is
> > running -- obviously it would start again at 0 when the server is
> restarted
> > or if the counter rolls over (unlikely for a 64 bit long).
> >
> > Below are two samples taken 60 seconds apart: the value decreased by
> > 2,954,369,012 between the two readings.
> >
> > reported_metric=[timestamp:1349476449, status:200,
> > request:[mbean:org.apache.cassandra.db:type=CompactionManager,
> > attribute:TotalBytesCompacted, type:read], value:7548675470069]
> >
> > previous_metric=[timestamp:1349476389, status:200,
> > request:[mbean:org.apache.cassandra.db:type=CompactionManager,
> > attribute:TotalBytesCompacted, type:read], value:7551629839081]
> >
> >
> > I briefly looked at the code for CompactionManager and a few related
> classes
> > and don't see anyplace that is performing subtraction explicitly;
> however,
> > there are many additions of signed long values that are not validated and
> > could conceivably contain a negative value thus causing the
> > totalBytesCompacted to decrease.  It's interesting to note that the all
> of
> > the differences I've seen so far are more than the overflow value of a
> > signed 32 bit value.  The OS (CentOS 5.7) and sun java vm (1.6.0_29) are
> > both 64 bit.  JNA is enabled.
> >
> > Is this expected and normal?  If so, what is the correct interpretation
> of
> > this metric?  I'm seeing the negatives values a few times per hour when
> > reading it once every 60 seconds.
> >
> > -Bryan
> >
>


-- 
Bryan Talbot
Architect / Platform team lead, Aeria Games and Entertainment
Silicon Valley | Berlin | Tokyo | Sao Paulo

--047d7b6766dc37812004cb910778
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I&#39;m attempting to plot how &quot;busy&quot; the node is doing compactio=
ns but there seems to only be a few metrics reported that might be suitable=
: CompletedTasks, PendingTasks, TotalBytesCompacted, TotalCompactionsComple=
ted.<div>
<br></div><div>It&#39;s not clear to me what the difference between Complet=
edTasks and TotalCompactionsCompleted is but I am plotting TotalCompactions=
Completed / sec as one metric; however, this rate is nearly always less tha=
n 1 and doesn&#39;t capture how much resources are used doing the compactio=
n. =C2=A0A compaction of 4 smallest SSTables counts the same as a compactio=
n of 4 largest SSTables but the cost is hugely different. =C2=A0Thus, I&#39=
;m also plotting TotalBytesCompacted / sec.</div>
<div><br></div><div>Since the TotalBytesCompacted value sometimes moves bac=
kwards I&#39;m not confident that it&#39;s reporting what it is meant to re=
port. =C2=A0The code and comments indicate that it should only be increment=
ed by the final size of the newly created SSTable or by the bytes-compacted=
-so-far for a larger compaction, so I don&#39;t see why it should be reason=
able for it to sometimes decrease.<br>
<div><br></div><div>How should the impact of compaction be measured if not =
by bytes compacted?</div><div><br></div><div>-Bryan</div><div><br><br><div =
class=3D"gmail_quote">On Sun, Oct 7, 2012 at 7:39 AM, Edward Capriolo <span=
 dir=3D"ltr">&lt;<a href=3D"mailto:edlinuxguru@gmail.com" target=3D"_blank"=
>edlinuxguru@gmail.com</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">I have not looked at this JMX object in a wh=
ile, however the<br>
compaction manager can support multiple threads. Also it moves from<br>
0-filesize each time it has to compact a set of files.<br>
<br>
That is more useful for showing current progress rather then lifetime histo=
ry.<br>
<div class=3D"HOEnZb"><div class=3D"h5"><br>
<br>
<br>
On Fri, Oct 5, 2012 at 7:27 PM, Bryan Talbot &lt;<a href=3D"mailto:btalbot@=
aeriagames.com">btalbot@aeriagames.com</a>&gt; wrote:<br>
&gt; I&#39;ve recently added compaction rate (in bytes / second) to my moni=
tors for<br>
&gt; cassandra and am seeing some odd values. =C2=A0I wasn&#39;t expecting =
the values for<br>
&gt; TotalBytesCompacted to sometimes decrease from one reading to the next=
. =C2=A0It<br>
&gt; seems that the value should be monotonically increasing while a server=
 is<br>
&gt; running -- obviously it would start again at 0 when the server is rest=
arted<br>
&gt; or if the counter rolls over (unlikely for a 64 bit long).<br>
&gt;<br>
&gt; Below are two samples taken 60 seconds apart: the value decreased by<b=
r>
&gt; 2,954,369,012 between the two readings.<br>
&gt;<br>
&gt; reported_metric=3D[timestamp:1349476449, status:200,<br>
&gt; request:[mbean:org.apache.cassandra.db:type=3DCompactionManager,<br>
&gt; attribute:TotalBytesCompacted, type:read], value:7548675470069]<br>
&gt;<br>
&gt; previous_metric=3D[timestamp:1349476389, status:200,<br>
&gt; request:[mbean:org.apache.cassandra.db:type=3DCompactionManager,<br>
&gt; attribute:TotalBytesCompacted, type:read], value:7551629839081]<br>
&gt;<br>
&gt;<br>
&gt; I briefly looked at the code for CompactionManager and a few related c=
lasses<br>
&gt; and don&#39;t see anyplace that is performing subtraction explicitly; =
however,<br>
&gt; there are many additions of signed long values that are not validated =
and<br>
&gt; could conceivably contain a negative value thus causing the<br>
&gt; totalBytesCompacted to decrease. =C2=A0It&#39;s interesting to note th=
at the all of<br>
&gt; the differences I&#39;ve seen so far are more than the overflow value =
of a<br>
&gt; signed 32 bit value. =C2=A0The OS (CentOS 5.7) and sun java vm (1.6.0_=
29) are<br>
&gt; both 64 bit. =C2=A0JNA is enabled.<br>
&gt;<br>
&gt; Is this expected and normal? =C2=A0If so, what is the correct interpre=
tation of<br>
&gt; this metric? =C2=A0I&#39;m seeing the negatives values a few times per=
 hour when<br>
&gt; reading it once every 60 seconds.<br>
&gt;<br>
&gt; -Bryan<br>
&gt;<br>
</div></div></blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>=
Bryan Talbot<div>Architect / Platform team lead, Aeria Games and Entertainm=
ent</div><div>Silicon Valley | Berlin | Tokyo | Sao Paulo</div><div><br>
</div><br>
</div></div>

--047d7b6766dc37812004cb910778--