kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jason Gustafson (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-1449) Extend wire protocol to allow CRC32C
Date Fri, 23 Sep 2016 22:56:20 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Jason Gustafson updated KAFKA-1449:
    Fix Version/s:     (was:

> Extend wire protocol to allow CRC32C
> ------------------------------------
>                 Key: KAFKA-1449
>                 URL: https://issues.apache.org/jira/browse/KAFKA-1449
>             Project: Kafka
>          Issue Type: Improvement
>          Components: consumer
>            Reporter: Albert Strasheim
>            Assignee: Neha Narkhede
>             Fix For:
> Howdy
> We are currently building out a number of Kafka consumers in Go, based on a patched version
of the Sarama library that Shopify released a while back.
> We have a reasonably fast serialization protocol (Cap'n Proto), a 10G network and lots
of cores. We have various consumers computing all kinds of aggregates on a reasonably high
volume access log stream (1.1e6 messages/sec peak, about 500-600 bytes per message uncompressed).
> When profiling our consumer, our single hottest function (until we disabled it), was
the CRC32 checksum validation, since the deserialization and aggregation in these consumers
is pretty cheap.
> We believe things could be improved by extending the wire protocol to support CRC-32C
(Castagnoli), since SSE 4.2 has an instruction to accelerate its calculation.
> https://en.wikipedia.org/wiki/SSE4#SSE4.2
> It might be hard to use from Java, but consumers written in most other languages will
benefit a lot.
> To give you an idea, here are some benchmarks for the Go CRC32 functions running on a
Intel(R) Core(TM) i7-3540M CPU @ 3.00GHz core:
> BenchmarkCrc32KB	 90196 ns/op 363.30 MB/s
> BenchmarkCrcCastagnoli32KB 3404 ns/op 9624.42 MB/s
> I believe BenchmarkCrc32 written in C would do about 600-700 MB/sec, and the CRC32-C
speed should be close to what one achieves in Go.
> (Met Todd and Clark at the meetup last night. Thanks for the great presentation!)

This message was sent by Atlassian JIRA

View raw message