kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ismael Juma (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (KAFKA-5465) FetchResponse v0 does not return any messages when max_bytes smaller than v2 message set
Date Sun, 18 Jun 2017 08:57:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16053101#comment-16053101
] 

Ismael Juma edited comment on KAFKA-5465 at 6/18/17 8:56 AM:
-------------------------------------------------------------

Jason has a PR in progress for this, but it's a little tricky and we'll probably need some
time to review and discuss. It's also late in the process, so we have to be particularly careful
not to introduce regressions.

It's worth pointing out that all of the following need to be true for this to happen:

1. Pre KIP-74 consumer (KIP-74 was introduced in 0.10.1.0)

2. Broker configured to use message format v2. Even though this is the default, the upgrade
instructions recommend that users only switch to the new message format after most consumers
have been upgraded to avoid the loss of zero-copy.

3. partition max bytes is less than the v2 record batch size. For this to happen with the
Java producer, the producer's max request size (default 1 MB for the Java producer) must be
larger than the consumer's max partition size (default 1 MB). Note that I didn't mention the
producer's batch size. The reason is that we allow a single record to be sent if it's larger
than the batch size, but smaller than max request size.

Given the above, it seems to me that users with default settings for the Java clients won't
run into the problem. It would be good to understand if it's the same for non Java clients
(cc [~edenhill [~dana.powers]). The workaround of setting the broker message format version
to v1 also seems straightforward (and it's what the upgrade instructions recommend if consumers
that don't support the new message format are still being used).

I will proceed with RC1. If we agree to include this, we can spin RC2.


was (Author: ijuma):
Jason has a PR in progress for this, but it's a little tricky and we'll probably need some
time to review and discuss. It's also late in the process, so we have to be particularly careful
not to introduce regressions.

It's worth pointing out that all of the following need to be true for this to happen:

1. Pre KIP-74 consumer (KIP-74 was introduced in 0.10.1.0)

2. Broker configured to use message format v2. Even though this is the default, the upgrade
instructions recommend that users only switch to the new message format after most consumers
have been upgraded to avoid the loss of zero-copy.

3. partition max bytes is less than the v2 record batch size. For this to happen with the
Java producer, the producer's max request size (default 1 MB for the Java producer) must be
larger than the consumer's max partition size (default 1 MB). Note that I didn't mention the
producer's batch size. The reason is that we allow a single record to be sent if it's larger
than the batch size, but smaller than max request size. Increasing the max request size to
be larger than the consumer's  max partition size is unsafe pre KIP-74 as one could send a
single message that's larger than what the consumer can fetch causing the latter to get stuck.

Given the above, it seems to me that users with default or recommended settings for the Java
clients won't run into the problem. It would be good to understand if it's the same for non
Java clients (cc [~edenhill [~dana.powers]). The workaround of setting the broker message
format version to v1 also seems straightforward (and it's what the upgrade instructions recommend
if consumers that don't support the new message format are still being used).

I will proceed with RC1. If we agree to include this, we can spin RC2.

> FetchResponse v0 does not return any messages when max_bytes smaller than v2 message
set 
> -----------------------------------------------------------------------------------------
>
>                 Key: KAFKA-5465
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5465
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.11.0.0
>            Reporter: Dana Powers
>            Assignee: Jason Gustafson
>            Priority: Blocker
>             Fix For: 0.11.0.1
>
>
> In prior releases, when consuming uncompressed messages, FetchResponse v0 will return
a message if it is smaller than the max_bytes sent in the FetchRequest. In 0.11.0.0 RC0, when
messages are stored as v2 internally, the response will be empty unless the full MessageSet
is smaller than max_bytes. In some configurations, this may cause old consumers to get stuck
on large messages where previously they were able to make progress one message at a time.
> For example, when I produce 10 5KB messages using ProduceRequest v0 and then attempt
FetchRequest v0 with partition max bytes = 6KB (larger than a single message but smaller than
all 10 messages together), I get an empty message set from 0.11.0.0. Previous brokers would
have returned a single message.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message