flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tzu-Li (Gordon) Tai (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-8001) Mark Kafka Consumer as idle if it doesn't have partitions
Date Tue, 07 Nov 2017 04:26:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-8001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16241466#comment-16241466
] 

Tzu-Li (Gordon) Tai edited comment on FLINK-8001 at 11/7/17 4:25 AM:
---------------------------------------------------------------------

I don't think 1.3 isn't affected by this though.
In 1.3, if there is no partitions to consume at the very beginning, the {{Long.MAX_VALUE}}
watermark is emitted as soon as the consumer starts, in which case the per-partition watermark
code paths are never relevant, I think.


was (Author: tzulitai):
I don't think 1.3 isn't affected by this though.
In 1.3, if there is no partitions to consume at the very beginning, the `Long.MAX_VALUE` watermark
is emitted as soon as the consumer starts, in which case the per-partition watermark code
paths are never relevant, I think.

> Mark Kafka Consumer as idle if it doesn't have partitions
> ---------------------------------------------------------
>
>                 Key: FLINK-8001
>                 URL: https://issues.apache.org/jira/browse/FLINK-8001
>             Project: Flink
>          Issue Type: Bug
>            Reporter: Aljoscha Krettek
>            Assignee: Tzu-Li (Gordon) Tai
>            Priority: Blocker
>             Fix For: 1.4.0, 1.3.3
>
>
> In Flink 1.3.x the Kafka Consumer will emit a {{Long.MAX_VALUE}} watermark if it has
zero partitions assigned. If this happens and other parallel instances of the Kafka Consumer
are marked as idle (which currently never happens by default but does happen in custom forks
of our Kafka code) this means that the watermark jumps to {{Long.MAX_VALUE}} downstream.
> In Flink 1.4.x this happens implicitly in the {{PeriodicWatermarkEmitter}} in {{AbstractFetcher}}
where the watermark is {{Long.MAX_VALUE}} if we don't have any partitions. This should be
changed to mark the source as idle instead, if we don't have any partitions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message