flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jakob Ericsson <jakob.erics...@gmail.com>
Subject Re: FlinkKafkaConsumer and multiple topics
Date Fri, 18 Sep 2015 12:21:30 GMT
Hit another problem. It is probably related to a topic that still exist in
zk but is not used anymore (therefore no partitions) or I want to start a
listener for a topic that hasn't yet been created. I would like it not to
crash.

Also, some funny Scala <-> Java

Exception in thread "main" java.lang.NoSuchMethodError:
kafka.common.ErrorMapping.InvalidTopicCode()S
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.getPartitionsForTopic(FlinkKafkaConsumer.java:619)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer.<init>(FlinkKafkaConsumer.java:280)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumer081.<init>(FlinkKafkaConsumer081.java:55)

On Fri, Sep 18, 2015 at 11:02 AM, Jakob Ericsson <jakob.ericsson@gmail.com>
wrote:

> That will work. We have some utility classes for exposing the ZK-info.
>
> On Fri, Sep 18, 2015 at 10:50 AM, Robert Metzger <rmetzger@apache.org>
> wrote:
>
>> Hi Jakob,
>>
>> currently, its not possible to subscribe to multiple topics with one
>> FlinkKafkaConsumer.
>>
>> So for now, you have to create a FKC for each topic .. so you'll end up
>> with 50 sources.
>>
>> As soon as Kafka releases the new consumer, it will support subscribing
>> to multiple topics (I think even with pattern support) and we can easily
>> expose the APIs also to the FlinkKafkaConsumer.
>> As you can see here:
>> https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan
>> Kafka has plans to release the new consumer API in October.
>> As soon as the new API is out, we'll support it.
>>
>> I hope this solution is okay for you. If not, please let me know ;)
>>
>>
>> Robert
>>
>> On Fri, Sep 18, 2015 at 10:43 AM, Jakob Ericsson <
>> jakob.ericsson@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Would it be possible to get the FlinkKafkaConsumer to support multiple
>>> topics, like a list?
>>>
>>> Or would it be better to instantiate one FlinkKafkaConsumers per topic
>>> and add as a source?
>>> We have about 40-50 topics to listen for one job.
>>> Or even better, supply a regexp pattern that defines the queues, this
>>> means that you have to do some queries against ZK to get information about
>>> topics.
>>>
>>> Kind regards,
>>> Jakob
>>>
>>
>>
>

Mime
View raw message