Mailing-List: contact user-help@cassandra.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@cassandra.apache.org
Received-SPF: pass (nike.apache.org: domain of jbellis@gmail.com designates
 74.125.82.44 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type:content-transfer-encoding;
        b=aTjOxyQQIskP4Bq71L/8l+Y4ieD1p8VbV/+4ibKNG2ij0Rxk7aCTqDj5Sb82c1xVz7
         QYuxGDi0Hb9rqpXJ1YVWV6mXhouQR/wSoihKzRfp5jBp5+Mm97iE6YHxUvGhyNVHeol3
         Eem3vHeN6vt1c97Cmz2VMpzWjYkXnnW8Oditk=
MIME-Version: 1.0
In-Reply-To: <20100521221344.GA33466@alumni.caltech.edu>
References: <20100520193329.GA31203@alumni.caltech.edu>
 <AANLkTinvTUzGR9jKNgjs3MeBIrwJmA4vjkEp4LliDTIP@mail.gmail.com>
	<20100521221344.GA33466@alumni.caltech.edu>
From: Jonathan Ellis <jbellis@gmail.com>
Date: Sat, 22 May 2010 07:12:59 -0700
Message-ID: <AANLkTikw1heZs0IkTmrtB88UlkgYVOSTnS-AFkIx-87p@mail.gmail.com>
Subject: Re: Compaction JMX Stats?
To: user@cassandra.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

they are null when there is no compaction in progress

On Fri, May 21, 2010 at 3:13 PM, Anthony Molinaro
<anthonym@alumni.caltech.edu> wrote:
> On Thu, May 20, 2010 at 02:11:23PM -0700, Jonathan Ellis wrote:
>> No, CM is not exposed to nodetool yet. =A0(You should really be putting
>> metrics into a real monitoring system rather than relying on nodetool.
>> =A0Some example munin plugins are at
>> http://github.com/jbellis/cassandra-munin-plugins, for instance.)
>>
>> CM also has BytesCompacted/BytesTotalInProgress.
>
> I actually do put metrics into a monitoring system, but often use nodetoo=
l
> to see what's happening as well. =A0However, the system I use to poll jmx
> is not finding BytesCompacted/BytesTotalInProgress (I mentioned the syste=
m
> we used in a previous email with some screenshots). =A0So I downloaded
> jmxquery.jar and ran it, which gave me this.
>
> % java -jar ./jmxquery.jar \
> =A0 --url=3Dservice:jmx:rmi:///jndi/rmi://localhost:32123/jmxrmi | \
> =A0 grep -i compaction
> org.apache.cassandra.db:type=3DCompactionManager%PendingTasks.value 0
> org.apache.cassandra.db:type=3DCompactionManager%MinimumCompactionThresho=
ld.value
> 4
> org.apache.cassandra.db:type=3DCompactionManager%MaximumCompactionThresho=
ld.value
> 32
> org.apache.cassandra.db:type=3DCompactionManager%ColumnFamilyInProgress.v=
alue null
> org.apache.cassandra.db:type=3DCompactionManager%BytesTotalInProgress.val=
ue null
> org.apache.cassandra.db:type=3DCompactionManager%BytesCompacted.value nul=
l
>
> Which explains why I never see it, our poller ignores 'null's
>
> So why are the Bytes* entries null? (I'm using 0.6.2 with the the first
> 4 bugs fixed 992, 1024 x 2, 1039). =A0Is this something which only works
> with trunk?
>
>> Backup in deserialize is likely to be (a) the read or write stage
>> being full so deserialize in turn backs up. =A0Anything that sucks up
>> enough CPU could also cause it, I suppose, but other than that
>> (generating lots of CPU load) there is nothing special about
>> compactions wrt deserialize.
>
> Is there some metric I could look at to determine if it's a? =A0CPU is
> usually at most 20% on these boxes, so I assume it's a causing the backup=
.
>
> -Anthony
>
>> On Thu, May 20, 2010 at 12:33 PM, Anthony Molinaro
>> <anthonym@alumni.caltech.edu> wrote:
>> > Hi,
>> >
>> > =A0In the 0.5.x series there was a COMPACTION-POOL which kept track of
>> > in process, pending and completed compactions. =A0With 0.6.x this seem=
s
>> > to have vanished and instead we only have the CompactionManager Pendin=
gTasks
>> > statistic. =A0Is there also a completed tasks somewhere? =A0Is there a=
ny
>> > way to determine via nodetool if a compaction or anticompaction is occ=
uring?
>> > I'm trying to figure out the cause of random spikes in the Message Des=
erializer
>> > Queue, and wanted to know if there was any correlation with compaction=
,
>> > but with only PendingTasks, I would have to be sampling continuously t=
o
>> > catch it, as pending is mostly zero.
>> >
>> > Are there other possible causes for a backup in Message Deserializer?
>> > My traffic patterns are pretty constant, I don't see spikes in the
>> > thread pool completed tasks, and often the rate is below the high wate=
r
>> > mark for rates.
>> >
>> > Thanks,
>> >
>> > -Anthony
>> >
>> > --
>> > ----------------------------------------------------------------------=
--
>> > Anthony Molinaro =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <=
anthonym@alumni.caltech.edu>
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 <ant=
honym@alumni.caltech.edu>
>


--=20
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com