kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sumant Tambe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-5621) The producer should retry expired batches when retries are enabled
Date Tue, 25 Jul 2017 16:55:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16100349#comment-16100349

Sumant Tambe commented on KAFKA-5621:

[~ijuma], [~apurva], It looks like this proposal will change the notional accumulator timeout
from request.timeout.ms (r.t.m) to  (r.t.m * retries). The bound on the send path of a record
from the point when send returns is up to linger.ms + r.t.m * retries + retries * (r.t.m.
+ retry.backoff.ms). So time of the send failure notification could potentially double as
two full retry cycles will be applied. 

I prefer an explicit name to the accumulator timeout (we proposed batch.expiry.ms) but I'm
well aware that adding new configs makes things harder for the enduser. I prefer to avoid
overloading configs to do multiple jobs. That's confusing too. Applying retries to accumulator
is not intuitive to me. It seems to jack up accumulator time by a multiple. 

Having said that, if there is strong agreement over not adding a new config, I'm fine with
the proposed trick here.

> The producer should retry expired batches when retries are enabled
> ------------------------------------------------------------------
>                 Key: KAFKA-5621
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5621
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Apurva Mehta
>             Fix For: 1.0.0
> Today, when a batch is expired in the accumulator, a {{TimeoutException}} is raised to
the user.
> It might be better the producer to retry the expired batch rather up to the configured
number of retries. This is more intuitive from the user's point of view. 
> Further the proposed behavior makes it easier for applications like mirror maker to provide
ordering guarantees even when batches expire. Today, they would resend the expired batch and
it would get added to the back of the queue, causing the output ordering to be different from
the input ordering.

This message was sent by Atlassian JIRA

View raw message