kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "aarti gupta (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3811) Introduce Kafka Streams metrics recording levels
Date Fri, 10 Jun 2016 16:37:20 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15324755#comment-15324755

aarti gupta commented on KAFKA-3811:

[~gfodor] Can you please share and outline the profiling and analysis you did around metrics

I just ran  the 'WordProcessorDemo' with actual data (continuously being published) on the
input stream and profiled the streams example using both Java mission control Flight recorder
and Yourkit profiler (evaluation version), but see a 5 % CPU overhead for the entire process.
How are you isolating the time taken to stamp metrics? 

> Introduce Kafka Streams metrics recording levels
> ------------------------------------------------
>                 Key: KAFKA-3811
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3811
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Greg Fodor
>            Assignee: aarti gupta
> Follow-up from the discussions here:
> https://github.com/apache/kafka/pull/1447
> https://issues.apache.org/jira/browse/KAFKA-3769
> The proposal is to introduce configuration to control the granularity/volumes of metrics
emitted by Kafka Streams jobs, since the per-record level metrics introduce non-trivial overhead
and are possibly less useful once a job has been optimized. 
> Proposal from guozhangwang:
> level0 (stream thread global): per-record process / punctuate latency, commit latency,
poll latency, etc
> level1 (per processor node, and per state store): IO latency, per-record .. latency,
forward throughput, etc.
> And by default we only turn on level0.

This message was sent by Atlassian JIRA

View raw message