incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Compaction JMX Stats?
Date Sat, 22 May 2010 14:12:59 GMT
they are null when there is no compaction in progress

On Fri, May 21, 2010 at 3:13 PM, Anthony Molinaro
<anthonym@alumni.caltech.edu> wrote:
> On Thu, May 20, 2010 at 02:11:23PM -0700, Jonathan Ellis wrote:
>> No, CM is not exposed to nodetool yet.  (You should really be putting
>> metrics into a real monitoring system rather than relying on nodetool.
>>  Some example munin plugins are at
>> http://github.com/jbellis/cassandra-munin-plugins, for instance.)
>>
>> CM also has BytesCompacted/BytesTotalInProgress.
>
> I actually do put metrics into a monitoring system, but often use nodetool
> to see what's happening as well.  However, the system I use to poll jmx
> is not finding BytesCompacted/BytesTotalInProgress (I mentioned the system
> we used in a previous email with some screenshots).  So I downloaded
> jmxquery.jar and ran it, which gave me this.
>
> % java -jar ./jmxquery.jar \
>   --url=service:jmx:rmi:///jndi/rmi://localhost:32123/jmxrmi | \
>   grep -i compaction
> org.apache.cassandra.db:type=CompactionManager%PendingTasks.value 0
> org.apache.cassandra.db:type=CompactionManager%MinimumCompactionThreshold.value
> 4
> org.apache.cassandra.db:type=CompactionManager%MaximumCompactionThreshold.value
> 32
> org.apache.cassandra.db:type=CompactionManager%ColumnFamilyInProgress.value null
> org.apache.cassandra.db:type=CompactionManager%BytesTotalInProgress.value null
> org.apache.cassandra.db:type=CompactionManager%BytesCompacted.value null
>
> Which explains why I never see it, our poller ignores 'null's
>
> So why are the Bytes* entries null? (I'm using 0.6.2 with the the first
> 4 bugs fixed 992, 1024 x 2, 1039).  Is this something which only works
> with trunk?
>
>> Backup in deserialize is likely to be (a) the read or write stage
>> being full so deserialize in turn backs up.  Anything that sucks up
>> enough CPU could also cause it, I suppose, but other than that
>> (generating lots of CPU load) there is nothing special about
>> compactions wrt deserialize.
>
> Is there some metric I could look at to determine if it's a?  CPU is
> usually at most 20% on these boxes, so I assume it's a causing the backup.
>
> -Anthony
>
>> On Thu, May 20, 2010 at 12:33 PM, Anthony Molinaro
>> <anthonym@alumni.caltech.edu> wrote:
>> > Hi,
>> >
>> >  In the 0.5.x series there was a COMPACTION-POOL which kept track of
>> > in process, pending and completed compactions.  With 0.6.x this seems
>> > to have vanished and instead we only have the CompactionManager PendingTasks
>> > statistic.  Is there also a completed tasks somewhere?  Is there any
>> > way to determine via nodetool if a compaction or anticompaction is occuring?
>> > I'm trying to figure out the cause of random spikes in the Message Deserializer
>> > Queue, and wanted to know if there was any correlation with compaction,
>> > but with only PendingTasks, I would have to be sampling continuously to
>> > catch it, as pending is mostly zero.
>> >
>> > Are there other possible causes for a backup in Message Deserializer?
>> > My traffic patterns are pretty constant, I don't see spikes in the
>> > thread pool completed tasks, and often the rate is below the high water
>> > mark for rates.
>> >
>> > Thanks,
>> >
>> > -Anthony
>> >
>> > --
>> > ------------------------------------------------------------------------
>> > Anthony Molinaro                           <anthonym@alumni.caltech.edu>
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
> --
> ------------------------------------------------------------------------
> Anthony Molinaro                           <anthonym@alumni.caltech.edu>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message