kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joe Stein (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-203) Improve Kafka internal metrics
Date Wed, 11 Jan 2012 01:02:39 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183760#comment-13183760

Joe Stein commented on KAFKA-203:

I have implemented Coda Hale's Metric package and have it running in production.

- good coverage for typical needs with not much extra effort once it is integrated
- ganglia context 
- graphite context (I don't use but it is a pro for someone)

- package causes a dependency
- the MetricsServlet is only JSON so some extra effort to have a version that also can be
better human readable (better for debugging something I found often)
- builds are 2.9.1 (I run kafka in 2.9.1 build with it and it works fine though porting it
back to 2.8 may get hair and tricky)


- since there is no compatible Scala version in maven we would basically have to checkout
or fork the code, build it and include it (or publish the build ourselves to maven). 
- we should have our own version of MetricsServlet ... if someone wants to implement the http
embedded jetty server it should be configurable which then makes the jetty jar not required

So I am +1 in using source from Coda Hale and keeping that layer as the layer for metrics
(we could check in the specific files we need/want to a package kafka.metrics and then make
changes as we need (license = ASL 2.0 https://github.com/codahale/metrics/blob/master/LICENSE)

I am comfortable with codahale/metrics source and I would contribute this by pulling the parts
in we are looking to use with an initial implementation for using them allowing (replacing
anything existing JMX as this would handle that and other context we include like the servlet,
ganglia, whatever else we want to make) others to-do so for other things moving forward. 

> Improve Kafka internal metrics
> ------------------------------
>                 Key: KAFKA-203
>                 URL: https://issues.apache.org/jira/browse/KAFKA-203
>             Project: Kafka
>          Issue Type: New Feature
>          Components: core
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
> Currently metrics in kafka are using old-school JMX directly. This makes adding metrics
a pain. It would be good to do one of the following:
> 1. Convert to Coda Hale's metrics package (https://github.com/codahale/metrics)
> 2. Write a simple metrics package
> The new metrics package should make metrics easier to add and work with and package up
the common logic of keeping windowed gauges, histograms, counters, etc. JMX should be just
one output of this.
> The advantage of the Coda Hale package is that it exists so we don't need to write it.
The downsides are (1) introduces another client dependency which causes conflicts, and (2)
seems a bit heavy on design. The good news is that the metrics-core package doesn't seem to
bring in a lot of dependencies which is nice, though the scala wrapper seems to want scala
2.9. I am also a little skeptical of the approach for histograms--it does sampling instead
of bucketing though that may be okay.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message