incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Edward Capriolo <edlinuxg...@gmail.com>
Subject Re: Cassandra Monitoring
Date Mon, 20 Dec 2010 16:40:39 GMT
On Sun, Dec 19, 2010 at 10:37 PM, Dave Viner <daveviner@gmail.com> wrote:
> Can you share the code for run_column_family_stores.sh ?
>
> On Sun, Dec 19, 2010 at 6:14 PM, Edward Capriolo <edlinuxguru@gmail.com>
> wrote:
>>
>> On Sun, Dec 19, 2010 at 2:01 PM, Ran Tavory <rantav@gmail.com> wrote:
>> > Mx4j is in process, same jvm, you just need to throw mx4j-tools.jar in
>> > the lib before you start Cassandra jmx-to-rest runs in a separate jvm.
>> >  It also has a nice useful HTML interface that you can look into any
>> > running host.
>> >
>> > On Sunday, December 19, 2010, Dave Viner <daveviner@gmail.com> wrote:
>> >> How does mx4j compare with the earlier jmx-to-rest bridge listed in the
>> >> operations page:
>> >> "JMX-to-REST bridge available
>> >> at http://code.google.com/p/polarrose-jmx-rest-bridge"
>> >>
>> >> ThanksDave Viner
>> >>
>> >>
>> >> On Sun, Dec 19, 2010 at 7:01 AM, Ran Tavory <rantav@gmail.com> wrote:
>> >> FYI, I just added an mx4j section to the bottom of this
>> >> page http://wiki.apache.org/cassandra/Operations
>> >>
>> >>
>> >> On Sun, Dec 19, 2010 at 4:30 PM, Jonathan Ellis <jbellis@gmail.com>
>> >> wrote:
>> >> mx4j? https://issues.apache.org/jira/browse/CASSANDRA-1068
>> >>
>> >>
>> >>
>> >>
>> >> On Sun, Dec 19, 2010 at 8:36 AM, Peter Schuller
>> >> <peter.schuller@infidyne.com> wrote:
>> >>> How / what are you monitoring? Best practices someone?
>> >>
>> >> I recently set up monitoring using the cassandra-munin-plugins
>> >> (https://github.com/jamesgolick/cassandra-munin-plugins). However, due
>> >> to various little details that wasn't too fun to integrate properly
>> >> with munin-node-configure and automated configuration management. A
>> >> problem is also the starting of a JVM for each use of jmxquery, which
>> >> can become a problem with many column families.
>> >>
>> >> I like your web server idea. Something persistent that can sit there
>> >> and do the JMX acrobatics, and expose something more easily consumed
>> >> for stuff like munin/zabbix/etc. It would be pretty nice to have that
>> >> out of the box with Cassandra, though I expect that would be
>> >> considered bloat. :)
>> >>
>> >> --
>> >> / Peter Schuller
>> >>
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of Riptano, the source for professional Cassandra support
>> >> http://riptano.com
>> >>
>> >>
>> >> --
>> >> /Ran
>> >>
>> >>
>> >>
>> >>
>> >
>> > --
>> > /Ran
>> >
>>
>> There is a lot of overhead on your monitoring station to kick up so
>> many JMX connections. There can also be nat/hostname problems for
>> remote JMX.
>>
>> My solution is to execute JMX over nagios remote plugin executor (NRPE).
>>
>> command[run_column_family_stores]=/usr/lib64/nagios/plugins/run_column_family_stores.sh
>> $ARG1$ $ARG2$ $ARG3$ $ARG4$ $ARG5$ $ARG6$
>>
>> Maybe not as fancy as a rest-jmx bridge, but solves most of the RMI
>> issues involved in pulling stats over JMX,
>
>

That script is just a wrapper:

For example we can have our NMS directly call the JMX fetch code like this:
java -cp /usr/lib64/nagios/plugins/cassandra-cacti-m6.jar
com.jointhegrid.m6.cassandra.CFStores
service:jmx:rmi:///jndi/rmi://<host>:<port>/jmxrmi <user> <pass>
org.apache.cassandra.db:columnfamily=<columnfamily>,keyspace=<keyspace>,type=ColumnFamilyStores

But as mentioned this puts a lot of pressure on the monitoring node to
open up all these JMX connections. With NRPE I can "farm" the requests
out over NRPE. Nodes end up executing their checks locally.

# cat /usr/lib64/nagios/plugins/run_column_family_stores.sh
java -cp /usr/lib64/nagios/plugins/cassandra-cacti-m6.jar
com.jointhegrid.m6.cassandra.CFStores
service:jmx:rmi:///jndi/rmi://${1}:${2}/jmxrmi ${3} ${4}
org.apache.cassandra.db:columnfamily=${5},keyspace=${6},type=ColumnFamilyStores

All the code is up here:
http://www.jointhegrid.com/cassandra/cassandra-cacti-m6.jsp
http://www.jointhegrid.com/svn/cassandra-cacti-m6/trunk/src/com/jointhegrid/m6/cassandra/CFStores.java

My main goal was to point out that you do not need REST bridges and
embedded web servers to run JMX checks remotely.

Mime
View raw message