flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-8181) Kafka producer's FlinkFixedPartitioner returns different partitions when a target topic is rescaled
Date Fri, 01 Dec 2017 08:29:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16274108#comment-16274108
] 

ASF GitHub Bot commented on FLINK-8181:
---------------------------------------

Github user tzulitai commented on a diff in the pull request:

    https://github.com/apache/flink/pull/5108#discussion_r154289435
  
    --- Diff: flink-connectors/flink-connector-kafka-base/src/main/java/org/apache/flink/streaming/connectors/kafka/partitioner/FlinkFixedPartitioner.java
---
    @@ -68,6 +78,13 @@ public int partition(T record, byte[] key, byte[] value, String targetTopic,
int
     			partitions != null && partitions.length > 0,
     			"Partitions of the target topic is empty.");
     
    -		return partitions[parallelInstanceId % partitions.length];
    +		if (topicToFixedPartition.containsKey(targetTopic)) {
    --- End diff --
    
    @aljoscha that is correct. I was hesitant in fixing that because it was a problem that
had always been there even in the previous version of the FlinkFixedPartitioner, but AFAIK
there hasn’t been complaints about it.
    
    We could also fix that for good now. What do you think?


> Kafka producer's FlinkFixedPartitioner returns different partitions when a target topic
is rescaled
> ---------------------------------------------------------------------------------------------------
>
>                 Key: FLINK-8181
>                 URL: https://issues.apache.org/jira/browse/FLINK-8181
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector
>    Affects Versions: 1.4.0, 1.5.0
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Blocker
>             Fix For: 1.4.0, 1.5.0
>
>
> On fixing FLINK-6288 and migrating the original {{FlinkFixedPartitioner}} to the new
partitioning API (commit 9ed9b68397b51bfd2b0f6e532212a82f771641bd), the {{FlinkFixedPartitioner}}
no longer returns identical target partitions once a target topic is rescaled.
> This results in a behavioral regression when the {{FlinkFixedPartitioner}} is used.
> The {{FlinkFixedPartitionerTest}} should also be strengthened to cover the target topic
rescaling case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message