kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Greg Fodor (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-3769) KStream job spending 60% of time writing metrics
Date Wed, 01 Jun 2016 21:37:59 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311179#comment-15311179

Greg Fodor commented on KAFKA-3769:

Thanks Jay!

Guozhang, what are your thoughts on instead of trying to reduce the granularity of the metrics,
potentially having a way to just disable the process/latency metrics collection? I'm still
pretty new to KStreams, and haven't used these metrics, but I'm guessing they will be used
for occasionally tuning the job against production data but not necessarily for operational
monitoring. (I could be wrong about this.) As such, it seems that you may want to just have
a switch you flip when you are running in production that will disable the metrics and maximize
the throughput of the job, and then turn it on selectively when you want to perform performance

> KStream job spending 60% of time writing metrics
> ------------------------------------------------
>                 Key: KAFKA-3769
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3769
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions:
>            Reporter: Greg Fodor
>            Assignee: Guozhang Wang
>            Priority: Critical
> I've been profiling a complex streams job, and found two major hotspots when writing
metrics, which take up about 60% of the CPU time of the job. (!) A PR is attached.

This message was sent by Atlassian JIRA

View raw message