flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From TechnoMage <mla...@technomage.com>
Subject Re: Kafka 0.11
Date Sun, 22 Apr 2018 16:48:57 GMT
The different versions of the connector correspond to different versions of Kafka.  If you
are using Kafka 0.8 use 0.8 connector, etc.  Versions of the connector after 0.10 support
exactly once delivery, versions prior to that only offer at least once delivery.

Kafka supports distributed processing through deployment of multiple brokers.  Each topic
can be divided into partitions and those allocated to brokers to get distributed processing
on a cluster.  I have not used its stream processing API yet, but I assume it runs within
the brokers working on partitions of a topic.

Cluster deployment for Flink and Kafka can be as a stand alone cluster (manual deployment
to a bunch of machines, via custom built AMI, etc), or through a cluster manager like Mesos,
Yarn, Kubernetes which will manage the work performed on the machines in a cluster.  I have
not seen any good tutorials on multi-machine deployments, there are a few suggesting how to
do so with kubernetes for Flink only, but none I have found for flink+kafka.  For my proof
of concept it was just easier to manually build out 4 machines.  The install for kafka and
flink are simple and getting java and maven on a base ubuntu image on AWS is quick, so I can
manually build out a machine in about 5 minutes.

Michael

> On Apr 22, 2018, at 2:22 AM, m@xi <makisntpap@gmail.com> wrote:
> 
> Hi Piotr!
> 
> In this page of the documentation [1] I can see the different versions of
> Kafka Connectors, but I am now learning about Kafka so some help would be
> valuable.
> 
> 1 -- Are 0.8, 0.9, 0.11 etc different version of the same thing or do they
> same thing? I mean does 0.11 offers everything the 0.8 already has?
> 
> 2 -- I would like to use Kafka Streams API in my flink cluster [2], which is
> used for standalone clusters if I am not mistaken, i.e. one node only by
> default.
> 
> 3 -- Can you give some hints and explain briefly about the cluster
> deployment with many machines? I mean what is Yarn, Mesos etc. I think they
> are "coordinators" of the cluster. But now that I would like to test my
> algorithm on a real cluster with several machines I would like some hints on
> which one should I use. What about Kubernetes and Docker [3] ?
> 
> Thanks a lot in advance!
> 
> Best,
> Max
> 
> [1] --
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/dev/connectors/kafka.html
> 
> [2] -- https://kafka.apache.org/documentation/streams/
> 
> [3] --
> https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/deployment/cluster_setup.html
> 
> 
> 
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/


Mime
View raw message