hadoop-zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kishore g <g.kish...@gmail.com>
Subject Re: monitoring zookeeper
Date Wed, 14 Apr 2010 23:50:50 GMT
Hi Travis,

We do monitor zookeeper using JMX. We have a simple code which does the

   - parse JMX output and convert the output into key value format. The
   nested properties are flattened.
   - Emit the key values using LWES[ http://www.lwes.org/] Api's at regular
   - The keys to be emitted can be configured via config file.

We have our own internal reporting framework which displays these metrics.
In order to differentiate between leader and follower we use separate keys


If the server is leader then rsf_mrl will be empty and vice versa. I can
provide the code to do this and you can probably change it to meet your
needs and enhance it to work for Ganglia. Let me know if this helps you.

Kishore G

On Wed, Apr 14, 2010 at 11:12 AM, Travis Crawford

> Hey zookeeper gurus -
> Are there any recommended ways for one to monitor zookeeper ensembles? I'm
> familiar with the four-letter words and that stats are published via JMX -
> I'm more interested in what people are doing with those stats.
> I'd like to publish the JMX stats to Ganglia, and this works well for the
> built-in stats. However, the zookeeper-specific names appear to be dynamic
> which causes issues when deciding what to publish. For example, the current
> mode (leader/follower) appears to only be accessible from the bean names,
> instead of looking at, say, a "mode" stat.
> org.apache.ZooKeeperService:name0=ReplicatedServer_id1,name1=replica.1,name2=Follower
> org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader
> The only way I've found to learn if replicas are up-to-date is looking at
> "synced" buried in followerInfo:
> $ java -jar cmdline-jmxclient-0.10.5.jar - localhost:8081
> org.apache.ZooKeeperService:name0=ReplicatedServer_id2,name1=replica.2,name2=Leader
> followerInfo
> 04/14/2010 18:06:06 +0000 org.archive.jmx.Client followerInfo:
> FollowerHandler Socket[addr=/,port=48104,localport=2888]
> tickOfLastAck:29793 synced?:true queuedPacketLength:0
> FollowerHandler Socket[addr=/,port=59599,localport=2888]
> tickOfLastAck:29793 synced?:true queuedPacketLength:0
> I don't mind writing a tool to parse the JMX output and publishing to
> Ganglia if needed, but it seems like a problem that may have already been
> solved and I'm curious what others are doing. The tool would basically take
> the zookeeper stats, normalize the names, and publish to a timeseries
> database.
> Is anyone already monitoring ZK in a way others might find useful?
> Thanks!
> Travis

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message