kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4254) Questionable handling of unknown partitions in KafkaProducer
Date Sat, 08 Oct 2016 08:02:21 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15557508#comment-15557508

ASF GitHub Bot commented on KAFKA-4254:

GitHub user kkonstantine opened a pull request:


    KAFKA-4254: Update producers metadata before failing on non-existent …


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/kkonstantine/kafka KAFKA-4254-Update-producers-metadata-before-failing-on-non-existent-partition

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1995
commit d6ea6b77b879ccd5b2d4980845b991052fd09b88
Author: Konstantine Karantasis <konstantine@confluent.io>
Date:   2016-10-08T08:00:00Z

    KAFKA-4254: Update producers metadata before failing on non-existent partition


> Questionable handling of unknown partitions in KafkaProducer
> ------------------------------------------------------------
>                 Key: KAFKA-4254
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4254
>             Project: Kafka
>          Issue Type: Bug
>          Components: producer 
>            Reporter: Jason Gustafson
>            Assignee: Konstantine Karantasis
>             Fix For:
> Currently the producer will raise an {{IllegalArgumentException}} if the user attempts
to write to a partition which has just been created. This is caused by the fact that the producer
does not attempt to refetch topic metadata in this case, which means that its check for partition
validity is based on stale metadata.
> If the topic for the partition did not already exist, it works fine because the producer
will block until it has metadata for the topic, so this case is primarily hit when the number
of partitions is dynamically increased. 
> A couple options to fix this that come to mind:
> 1. We could treat unknown partitions just as we do unknown topics. If the partition doesn't
exist, we refetch metadata and try again (timing out when max.block.ms is reached).
> 2. We can at least throw a more specific exception so that users can handle the error.
Raising {{IllegalArgumentException}} is not helpful in practice because it can also be caused
by other error.s
> My inclination is to do the first one since the producer seems incorrect to tell the
user that the partition is invalid.

This message was sent by Atlassian JIRA

View raw message