kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "aarti gupta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3811) Introduce Kafka Streams metrics recording levels
Date Mon, 13 Jun 2016 05:51:21 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326855#comment-15326855

aarti gupta commented on KAFKA-3811:

Yes indeed, as per http://www.brendangregg.com/blog/2014-06-09/java-cpu-sampling-using-hprof.html

Tried to do something similar using simpleBenchmark (on a different fix) and yourkit profiler
as [~gfodor] mentions below, here https://github.com/apache/kafka/pull/1446#issuecomment-225488213,
but not convinced that the results mean anything conclusive.
thoughts/suggestions on a repeatable/consistent toolset 

Like the idea of 3>>Add a general purpose feature to the metrics library and use it
across the producer, consumer, and streams.

But before we refactor the existing library want a reproducible test, any suggestions on a
sceanrio other than SimpleBenchmark?

> Introduce Kafka Streams metrics recording levels
> ------------------------------------------------
>                 Key: KAFKA-3811
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3811
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Greg Fodor
>            Assignee: aarti gupta
>         Attachments: Muon-Snapshot.zip, Muon-latency.zip, screenshot-1.png, screenshot-latency.png
> Follow-up from the discussions here:
> https://github.com/apache/kafka/pull/1447
> https://issues.apache.org/jira/browse/KAFKA-3769
> The proposal is to introduce configuration to control the granularity/volumes of metrics
emitted by Kafka Streams jobs, since the per-record level metrics introduce non-trivial overhead
and are possibly less useful once a job has been optimized. 
> Proposal from guozhangwang:
> level0 (stream thread global): per-record process / punctuate latency, commit latency,
poll latency, etc
> level1 (per processor node, and per state store): IO latency, per-record .. latency,
forward throughput, etc.
> And by default we only turn on level0.

This message was sent by Atlassian JIRA

View raw message