flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Ericsson <jakob.erics...@gmail.com>
Subject Re: FlinkKafkaConsumer and multiple topics
Date Fri, 18 Sep 2015 12:21:30 GMT
Hit another problem. It is probably related to a topic that still exist in
zk but is not used anymore (therefore no partitions) or I want to start a
listener for a topic that hasn't yet been created. I would like it not to

Also, some funny Scala <-> Java

Exception in thread "main" java.lang.NoSuchMethodError:

On Fri, Sep 18, 2015 at 11:02 AM, Jakob Ericsson <jakob.ericsson@gmail.com>

> That will work. We have some utility classes for exposing the ZK-info.
> On Fri, Sep 18, 2015 at 10:50 AM, Robert Metzger <rmetzger@apache.org>
> wrote:
>> Hi Jakob,
>> currently, its not possible to subscribe to multiple topics with one
>> FlinkKafkaConsumer.
>> So for now, you have to create a FKC for each topic .. so you'll end up
>> with 50 sources.
>> As soon as Kafka releases the new consumer, it will support subscribing
>> to multiple topics (I think even with pattern support) and we can easily
>> expose the APIs also to the FlinkKafkaConsumer.
>> As you can see here:
>> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
>> Kafka has plans to release the new consumer API in October.
>> As soon as the new API is out, we'll support it.
>> I hope this solution is okay for you. If not, please let me know ;)
>> Robert
>> On Fri, Sep 18, 2015 at 10:43 AM, Jakob Ericsson <
>> jakob.ericsson@gmail.com> wrote:
>>> Hi,
>>> Would it be possible to get the FlinkKafkaConsumer to support multiple
>>> topics, like a list?
>>> Or would it be better to instantiate one FlinkKafkaConsumers per topic
>>> and add as a source?
>>> We have about 40-50 topics to listen for one job.
>>> Or even better, supply a regexp pattern that defines the queues, this
>>> means that you have to do some queries against ZK to get information about
>>> topics.
>>> Kind regards,
>>> Jakob

View raw message