flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From AndreaKinn <kinn6...@hotmail.it>
Subject Assigning operators to slots
Date Fri, 08 Sep 2017 14:32:10 GMT
Hi, firstly excuse me for the long post.
I already read the documentation about parallelism, slots and the API about
it but I still have some doubts about practical implementations of them.
My program is composed essentially by three operations:

- get data from a kafka source
- perform a machine learning operator on the retrieved stream
- push out data to a cassandra sink

I'd like to investigate and trying implement them in two different
situations:


1) FIRST ONE

Imagine I have a single dual core physical node and suppose I set
NumberOfTaskSlot = NumberOfCore (As suggested by the doc). 

<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t985/tonabble.png>


I suppose I can divide in a fixed way the operations into slots as described
in the figure. Is this possible?
Can I do that using slotSharingGroup(groupname) method ? Or have I to use
startNewChain() between the operator?
Example:

*DataStream<MyEvent> stream = env
				.addSource(new FlinkKafkaConsumer010<>(TOPIC, new CustomDeserializer(),
properties))
				.assignTimestampsAndWatermarks(new CustomTimestampExtractor())
				.map(...)
				.slotSharingGroup("source");*

or

*DataStream<MyEvent> stream = env
				.addSource(new FlinkKafkaConsumer010<>(TOPIC, new CustomDeserializer(),
properties))
.startNewChain()
				.assignTimestampsAndWatermarks(new CustomTimestampExtractor())
.startNewChain()
				.map(...);
				*


2) SECOND ONE

Imagine I have 3 dual core physical nodes.

<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/t985/tonabble2.png>


I suppose I can reserve one physical NODE for each operation. Is this
possible?
In this case honestly I don't know how to implement that at level code.
Moreover, I don't know if it would has sense set NumberTaskSlot =
NumberOfCores or to leave this option to Flink's choice.








--
Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Mime
View raw message