flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Metzger <rmetz...@apache.org>
Subject Re: FlinkKafkaConsumer bootstrap.servers vs. broker hosts
Date Wed, 14 Oct 2015 14:46:06 GMT
Hi Juho,

sorry for the late reply, I was busy with Flink Forward :)

The Flink Kafka Consumer needs both addresses.
Kafka uses the bootstrap servers to connect to the brokers to consume
messages.
The Zookeeper connection is used to commit the offsets of the consumer
group once a state snapshot in Flink has been completed.

Our consumers are pretty minimalistic because we are waiting for the Kafka
project to finish their new consumer API [2]. It seems that its a matter of
weeks for the Kafka project to release the new consumer [1].
With the new consumer API from kafka, we can actually give Flink users a
lot more features, for example subscribing to multiple topics with one
source instance, committing offsets to brokers, etc.
I think the new consumer will not need both bootstrap servers and zookeeper.


Is the problem you're reporting preventing you from using the KafkaConsumer
in production?
If so, can you tell me when exactly the data consumption is failing? (maybe
with some logs / stacktrace?)

If this contains confidential information, you can also send me a private
mail.


Regards,
Robert




[1] https://twitter.com/gwenshap/status/653718350648897536
[2]
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design


On Tue, Oct 13, 2015 at 3:29 PM, Juho Autio <juho.autio@rovio.com> wrote:

> The FlinkKafkaConsumer takes the two arguments --bootstrap.servers (kafka
> servers) and --zookeeper.connect (zookeeper servers). Then it seems to
> resolve Kafka brokers from zookeeper, and uses those host names to consume
> kafka. But it also connects to the given bootstrap servers to fetch some
> metadata, it seems.
>
> This is problematic when zookeeper uses internal kafka hostnames –
> consuming
> won't work when those resolved kafka hosts can't be reached.
>
> Could the consumer be changed to
> - respect the provided kafka hosts / ips and not use host names resolved
> from zookeeper
> and optionally
> - not require bootstrap.servers argument at all, just resolve broker host
> names from zookeeper and use those as "bootstrap servers", too?
>
> Is the concept of bootstrap server something entirely else, or what am I
> missing here?
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkKafkaConsumer-bootstrap-servers-vs-broker-hosts-tp3109.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

Mime
View raw message