kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Jorgensen (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-4141) 2x increase in cpu usage on new Producer API
Date Wed, 07 Sep 2016 21:13:21 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-4141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Andrew Jorgensen updated KAFKA-4141:
    Summary: 2x increase in cpu usage on new Producer API  (was: 2x increase in cp usage on
new Producer API)

> 2x increase in cpu usage on new Producer API
> --------------------------------------------
>                 Key: KAFKA-4141
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4141
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Andrew Jorgensen
> We are seeing about a 2x increase in CPU usage for the new kafka producer compared to
the producer. We are currently using gzip compression.
> We recently upgraded our kafka server and producer to from and noticed
that the cpu usage for the new producer had increased pretty significantly compared to the
old producer. This has caused us to need more resources to do the same amount of work we were
doing before. I did some quick profiling and it looks like during sending half of the cpu
cycles are sped in org.apache.kafka.common.record.Compressor.putRecord and the other half
is in org.apache.kafka.common.record.Record.computeChecksum (both are around 5.8% of the cpu
cycles for that method). I know its not apples to apples but the old producer did not seem
to have this overhead or at least it was greatly reduced.
> Is this a known performance degradation compared to the old producer? 
> Here is the trace from the old producer:
> !http://imgur.com/1xS34Dl.jpg!
> New producer:
> !http://imgur.com/0w0G5b1.jpg!

This message was sent by Atlassian JIRA

View raw message