flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7195) FlinkKafkaConsumer should not respect fetched partitions to filter restored partition states
Date Tue, 18 Jul 2017 09:56:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16091362#comment-16091362
] 

ASF GitHub Bot commented on FLINK-7195:
---------------------------------------

Github user aljoscha commented on the issue:

    https://github.com/apache/flink/pull/4344
  
    Ah, I thought you meant something else. Because this is pretty much the bug this PR is
trying to solve, right? 😅 


> FlinkKafkaConsumer should not respect fetched partitions to filter restored partition
states
> --------------------------------------------------------------------------------------------
>
>                 Key: FLINK-7195
>                 URL: https://issues.apache.org/jira/browse/FLINK-7195
>             Project: Flink
>          Issue Type: Bug
>          Components: Kafka Connector
>    Affects Versions: 1.3.1
>            Reporter: Tzu-Li (Gordon) Tai
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Blocker
>             Fix For: 1.3.2
>
>
> This issue is a re-appearance of FLINK-6006. On restore, we should not respect any fetched
partitions list from Kafka and perform any filtering of the restored partition states. There
are corner cases where, due to Kafka broker downtime, some partitions may be missing in the
fetched partition list. To be more precise, we actually should not require fetching partitions
on restore.
> We've stepped on our own foot again and reintroduced this bug in https://github.com/apache/flink/pull/3378/commits/ed68fedbe90db03823d75a020510ad3c344fa73e.
The previous test for this behavior was too implementation specific, and therefore the leak
in catching this on different internal implementations.
> We should have a proper unit test for this that does not rely on the internal implementations
and test only on public abstractions of {{FlinkKafkaConsumerBase}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message