kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jun Rao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-203) Improve Kafka internal metrics
Date Tue, 04 Sep 2012 17:04:08 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jun Rao updated KAFKA-203:

    Attachment: kafka-203_v1.patch

Attach patch v1. 

Patch overview:
1. Added general support to collect time breakdown (queueTime, localTime, remoteTime, sendTime,
totalTime) for all types of requests. Need to refactor RequestChannel.Request a bit to include
deserialized request object.
2. removed some metrics in DelayedRequestMetrics since they are now covered by #1.
3. Fixed Pool.getMaybePut() to make sure that the new object is only created once.
4. Converted all existing metrics to use coda hale and added some new metrics.

The list of new and converted metrics is the following.
Server side:
A. Requests 
for each request type: 
A1. requestRate (meter, total)
A2. queueTime (hist, total)
A3. localTime (hist, total)
A4. remoteTime (hist, total)
A5. sendTime (hist, total)
A6. totalTime (hist, total)
For Fetch/Produce
A7. produceFailureRate (meter, topic/total)
A8. fetchFailureRate (meter, topic/total)
A9. messagesInRate (meter, topic/total)
A10. messagesOutRate (meter, topic/total)
A11. messagesBytesInRate (meter, topic/total)
A12. messagesBytesOutRate (meter, topic/total)
A13. requestQueueSize (gauge, total)

B. Log:
B1. logFlushTime (timer, total)
B2. logSegments (gauge, per log)
B3. logEndOffset (gauge, per log)

C. Purgatory:
C1. expiredRequestMeter (meter, partition/total)

C2. expiredRequestMeter (meter, follower/non-follower)

C3. delayedRequests (gauge, Fetch/Produce)

D. ReplicaManager:
D1. leaderPartitionCounts (gauge, total)
D2. underReplicatedPartitionCounts (|ISR| < replication factor, gauge, total)
D3. ISRExpandRate (meter, partition/total)
D4. ISRShrinkRate (meter, partition/total)

E. Controller:
E1. controllerActiveCount (gauge, total)

F. Producer:
F1. messageRate (meter, topic/total)
F2. byteRate (meter, topic/total)
F3. droppedEventRate (meter, total)
F4. producerQueueSize (gauge, per send thread)
F5. requestSizeHist (hist, total)
F6. requestTimeAndRate (timer, total)
F7. resendRate (meter, total)
F8. failedSendRate (meter, total)

G. Consumer:
G1. messageRate (meter, topic/total)
G2. byteRate (meter, topic/total)
G3. requestSizeHist (hist, total)
G4. requestTimeAndRate (timer, total)
G5. lagInBytes (gauge, partition)
> Improve Kafka internal metrics
> ------------------------------
>                 Key: KAFKA-203
>                 URL: https://issues.apache.org/jira/browse/KAFKA-203
>             Project: Kafka
>          Issue Type: New Feature
>          Components: core
>    Affects Versions: 0.8
>            Reporter: Jay Kreps
>            Assignee: Jun Rao
>              Labels: tools
>         Attachments: kafka-203_v1.patch
> Currently metrics in kafka are using old-school JMX directly. This makes adding metrics
a pain. It would be good to do one of the following:
> 1. Convert to Coda Hale's metrics package (https://github.com/codahale/metrics)
> 2. Write a simple metrics package
> The new metrics package should make metrics easier to add and work with and package up
the common logic of keeping windowed gauges, histograms, counters, etc. JMX should be just
one output of this.
> The advantage of the Coda Hale package is that it exists so we don't need to write it.
The downsides are (1) introduces another client dependency which causes conflicts, and (2)
seems a bit heavy on design. The good news is that the metrics-core package doesn't seem to
bring in a lot of dependencies which is nice, though the scala wrapper seems to want scala
2.9. I am also a little skeptical of the approach for histograms--it does sampling instead
of bucketing though that may be okay.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message