incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anthony Molinaro <antho...@alumni.caltech.edu>
Subject Re: Compaction JMX Stats?
Date Fri, 21 May 2010 22:13:44 GMT
On Thu, May 20, 2010 at 02:11:23PM -0700, Jonathan Ellis wrote:
> No, CM is not exposed to nodetool yet.  (You should really be putting
> metrics into a real monitoring system rather than relying on nodetool.
>  Some example munin plugins are at
> http://github.com/jbellis/cassandra-munin-plugins, for instance.)
>
> CM also has BytesCompacted/BytesTotalInProgress.

I actually do put metrics into a monitoring system, but often use nodetool
to see what's happening as well.  However, the system I use to poll jmx
is not finding BytesCompacted/BytesTotalInProgress (I mentioned the system
we used in a previous email with some screenshots).  So I downloaded
jmxquery.jar and ran it, which gave me this.

% java -jar ./jmxquery.jar \
   --url=service:jmx:rmi:///jndi/rmi://localhost:32123/jmxrmi | \
   grep -i compaction
org.apache.cassandra.db:type=CompactionManager%PendingTasks.value 0
org.apache.cassandra.db:type=CompactionManager%MinimumCompactionThreshold.value
4
org.apache.cassandra.db:type=CompactionManager%MaximumCompactionThreshold.value
32
org.apache.cassandra.db:type=CompactionManager%ColumnFamilyInProgress.value null
org.apache.cassandra.db:type=CompactionManager%BytesTotalInProgress.value null
org.apache.cassandra.db:type=CompactionManager%BytesCompacted.value null

Which explains why I never see it, our poller ignores 'null's

So why are the Bytes* entries null? (I'm using 0.6.2 with the the first
4 bugs fixed 992, 1024 x 2, 1039).  Is this something which only works
with trunk?

> Backup in deserialize is likely to be (a) the read or write stage
> being full so deserialize in turn backs up.  Anything that sucks up
> enough CPU could also cause it, I suppose, but other than that
> (generating lots of CPU load) there is nothing special about
> compactions wrt deserialize.

Is there some metric I could look at to determine if it's a?  CPU is
usually at most 20% on these boxes, so I assume it's a causing the backup.

-Anthony

> On Thu, May 20, 2010 at 12:33 PM, Anthony Molinaro
> <anthonym@alumni.caltech.edu> wrote:
> > Hi,
> >
> >  In the 0.5.x series there was a COMPACTION-POOL which kept track of
> > in process, pending and completed compactions.  With 0.6.x this seems
> > to have vanished and instead we only have the CompactionManager PendingTasks
> > statistic.  Is there also a completed tasks somewhere?  Is there any
> > way to determine via nodetool if a compaction or anticompaction is occuring?
> > I'm trying to figure out the cause of random spikes in the Message Deserializer
> > Queue, and wanted to know if there was any correlation with compaction,
> > but with only PendingTasks, I would have to be sampling continuously to
> > catch it, as pending is mostly zero.
> >
> > Are there other possible causes for a backup in Message Deserializer?
> > My traffic patterns are pretty constant, I don't see spikes in the
> > thread pool completed tasks, and often the rate is below the high water
> > mark for rates.
> >
> > Thanks,
> >
> > -Anthony
> >
> > --
> > ------------------------------------------------------------------------
> > Anthony Molinaro                           <anthonym@alumni.caltech.edu>
> >
> 
> 
> 
> -- 
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com

-- 
------------------------------------------------------------------------
Anthony Molinaro                           <anthonym@alumni.caltech.edu>

Mime
View raw message