kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Onur Karaman (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-3718) propagate all KafkaConfig __consumer_offsets configs to OffsetConfig instantiation
Date Thu, 19 May 2016 01:14:12 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-3718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Onur Karaman updated KAFKA-3718:
--------------------------------
    Fix Version/s: 0.10.0.0

> propagate all KafkaConfig __consumer_offsets configs to OffsetConfig instantiation
> ----------------------------------------------------------------------------------
>
>                 Key: KAFKA-3718
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3718
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Onur Karaman
>            Assignee: Onur Karaman
>             Fix For: 0.10.0.0
>
>
> Kafka has two configurable compression codecs: the one used by the client (source codec)
and the one finally used when storing into the log (target codec). The target codec defaults
to KafkaConfig.compressionType and can be dynamically configured through zookeeper.
> The GroupCoordinator appends group membership information into the __consumer_offsets
topic by:
> 1. making a message with group membership information
> 2. making a MessageSet with the single message compressed with the source codec
> 3. doing a log.append on the MessageSet
> Without this patch, KafkaConfig.offsetsTopicCompressionCodec doesn't get propagated to
OffsetConfig instantiation, so GroupMetadataManager uses a source codec of NoCompressionCodec
when making the MessageSet. Let's say we have enough group information such that the message
formed exceeds KafkaConfig.messageMaxBytes before compression but would fall below the threshold
after compression using our source codec. Even if we had dynamically configured __consumer_offsets
with our favorite compression codec, the log.append will throw RecordTooLargeException during
analyzeAndValidateMessageSet since the message was unexpectedly uncompressed instead of having
been compressed with the source codec defined by KafkaConfig.offsetsTopicCompressionCodec.
> NOTE: even after this issue is resolved, preliminary tests show that LinkedIn will still
hit RecordTooLargeException with large groups that consume many topics (like MirrorMakers
with wildcard consumption of .*) since fully expanded subscription and assignment state for
each member is put into a single record. But this is a first step in the right direction.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message