kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Xavier Léauté (JIRA) <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5150) LZ4 decompression is 4-5x slower than Snappy on small batches / messages
Date Fri, 05 May 2017 22:28:04 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15999059#comment-15999059
] 

Xavier Léauté commented on KAFKA-5150:
--------------------------------------

I also have a versions of the same patch for 0.10.2.x in case someone is interested, or if
we decide to do another 0.10.2.x release.

https://github.com/apache/kafka/compare/0.10.2...xvrl:kafka-5150-0.10?expand=1

> LZ4 decompression is 4-5x slower than Snappy on small batches / messages
> ------------------------------------------------------------------------
>
>                 Key: KAFKA-5150
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5150
>             Project: Kafka
>          Issue Type: Bug
>          Components: consumer
>    Affects Versions: 0.8.2.2, 0.9.0.1, 0.11.0.0, 0.10.2.1
>            Reporter: Xavier Léauté
>            Assignee: Xavier Léauté
>             Fix For: 0.11.0.0
>
>
> I benchmarked RecordsIteratorDeepRecordsIterator instantiation on small batch sizes with
small messages after observing some performance bottlenecks in the consumer. 
> For batch sizes of 1 with messages of 100 bytes, LZ4 heavily underperforms compared to
Snappy (see benchmark below). Most of our time is currently spent allocating memory blocks
in KafkaLZ4BlockInputStream, due to the fact that we default to larger 64kB block sizes. Some
quick testing shows we could improve performance by almost an order of magnitude for small
batches and messages if we re-used buffers between instantiations of the input stream.
> [Benchmark Code|https://github.com/xvrl/kafka/blob/small-batch-lz4-benchmark/clients/src/test/java/org/apache/kafka/common/record/DeepRecordsIteratorBenchmark.java#L86]
> {code}
> Benchmark                                              (compressionType)  (messageSize)
  Mode  Cnt       Score       Error  Units
> DeepRecordsIteratorBenchmark.measureSingleMessage                    LZ4            100
 thrpt   20   84802.279 ±  1983.847  ops/s
> DeepRecordsIteratorBenchmark.measureSingleMessage                 SNAPPY            100
 thrpt   20  407585.747 ±  9877.073  ops/s
> DeepRecordsIteratorBenchmark.measureSingleMessage                   NONE            100
 thrpt   20  579141.634 ± 18482.093  ops/s
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message