flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shengkai Fang (Jira)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-18449) Make topic discovery and partition discovery configurable for FlinkKafkaConsumer in Table API
Date Wed, 01 Jul 2020 07:27:00 GMT

    [ https://issues.apache.org/jira/browse/FLINK-18449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17149183#comment-17149183
] 

Shengkai Fang commented on FLINK-18449:
---------------------------------------

It's my pleasure to work on this.

For the first question, we can use format {{'[topic-1, topic-2]'}} and format {{'topic'}}
to distinguish static topic and topic list under the same key {{'topic'}}. But I think we
still need key, such as {{'topic-regex'}} to specify use of regex. Because some characters
in regex have special meaning, such as '[' and ']'. Therefore, we would better to use another
key to allow users to express their need for regex.

For the second question, maybe we can use '{{topic-partition-discovery.interval-millis}}'.

> Make topic discovery and partition discovery configurable for FlinkKafkaConsumer in Table
API
> ---------------------------------------------------------------------------------------------
>
>                 Key: FLINK-18449
>                 URL: https://issues.apache.org/jira/browse/FLINK-18449
>             Project: Flink
>          Issue Type: Improvement
>          Components: Connectors / Kafka, Table SQL / API
>    Affects Versions: 1.11.0
>            Reporter: Shengkai Fang
>            Priority: Major
>
> In streaming api, we can use regex to find topic and enable partiton discovery by setting
non-negative value for `{{flink.partition-discovery.interval-millis}}`. However, it's not
work in table api. I think we can add options such as 'topic-regex' and '{{partition-discovery.interval-millis}}'
in WITH block for users.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message