kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5032) Think through implications of max.message.size affecting record batches in message format V2
Date Mon, 19 Jun 2017 21:24:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16054763#comment-16054763

ASF GitHub Bot commented on KAFKA-5032:

GitHub user apurvam opened a pull request:


    KAFKA-5032: Update the docs for message size configs across the board

    Before 0.11, we used to have limits for maximum message size on the producer, broker,
and consumer side.
    From 0.11 onward, these limits apply to record batches as a whole. This patch updates
the documentation of the configs to make this explicit. 
    A separate patch will have more extensive upgrade notes to tie all the changes together
in one narrative. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apurvam/kafka KAFKA-5032-message-size-docs

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3374
commit b8e1379a54d21141a22694b2aa6d422709bfb89f
Author: Apurva Mehta <apurva@confluent.io>
Date:   2017-06-19T21:20:03Z

    Change references to 'message' in the size options to 'record batch', since everything
is written and read in batches in the current version.


> Think through implications of max.message.size affecting record batches in message format
> --------------------------------------------------------------------------------------------
>                 Key: KAFKA-5032
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5032
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: clients, core, producer 
>            Reporter: Ismael Juma
>            Assignee: Apurva Mehta
>            Priority: Critical
>              Labels: documentation, exactly-once
>             Fix For:
> It's worth noting that the new behaviour for uncompressed messages is the same as the
existing behaviour for compressed messages.
> A few things to think about:
> 1. Do the producer settings max.request.size and batch.size still make sense and do we
need to update the documentation? My conclusion is that things are still fine, but we may
need to revise the docs.
> 2. (Seems like we don't need to do this) Consider changing default max message set size
to include record batch overhead. This is currently defined as:
> {code}
> val MessageMaxBytes = 1000000 + MessageSet.LogOverhead
> {code}
> We should consider changing it to (I haven't thought it through though):
> {code}
> val MessageMaxBytes = 1000000 + DefaultRecordBatch.RECORD_BATCH_OVERHEAD
> {code}
> 3. When a record batch is too large, we throw RecordTooLargeException, which is confusing
because there's also a RecordBatchTooLargeException. We should consider renaming these exceptions
to make the behaviour clearer.
> 4. We should consider deprecating max.message.bytes (server config) and message.max.bytes
(topic config) in favour of configs that make it clear that we are talking about record batches
instead of individual messages.
> Part of the work in this JIRA is working out what should be done for and what
can be done later.

This message was sent by Atlassian JIRA

View raw message