flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8181) Kafka producer's FlinkFixedPartitioner returns different partitions when a target topic is rescaled
Date Fri, 01 Dec 2017 07:44:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274073#comment-16274073
] 

ASF GitHub Bot commented on FLINK-8181:
---------------------------------------

Github user aljoscha commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5108#discussion_r154283216
  
    --- Diff: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/partitioner/FlinkFixedPartitioner.java
---
    @@ -68,6 +78,13 @@ public int partition(T record, byte[] key, byte[] value, String targetTopic,
int
     			partitions != null && partitions.length > 0,
     			"Partitions of the target topic is empty.");
     
    -		return partitions[parallelInstanceId % partitions.length];
    +		if (topicToFixedPartition.containsKey(targetTopic)) {
    --- End diff --
    
    This will only work if we don't restart the job, right? Because the state of the partitioner
is lost when we have a failure.


> Kafka producer's FlinkFixedPartitioner returns different partitions when a target topic
is rescaled
> ---------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-8181
>                 URL: https://issues.apache.org/jira/browse/FLINK-8181
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector
>    Affects Versions: 1.4.0, 1.5.0
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Blocker
>             Fix For: 1.4.0, 1.5.0
>
>
> On fixing FLINK-6288 and migrating the original {{FlinkFixedPartitioner}} to the new
partitioning API (commit 9ed9b68397b51bfd2b0f6e532212a82f771641bd), the {{FlinkFixedPartitioner}}
no longer returns identical target partitions once a target topic is rescaled.
> This results in a behavioral regression when the {{FlinkFixedPartitioner}} is used.
> The {{FlinkFixedPartitionerTest}} should also be strengthened to cover the target topic
rescaling case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message