flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From sohimankotia <sohimanko...@gmail.com>
Subject Partitioner is spending around 2 to 4 minutes while pushing data to next operator
Date Thu, 22 Jun 2017 09:58:31 GMT
I have a execution flow (Streaming Job) with parallelism 1.

 source -> map -> partitioner -> flatmap -> sink 

Since adding partitioner will start new thread but partitioner is spending
average of 2 to 4 minutes while moving data from map to flat map .

For more details about this  : 
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Using-Custom-Partitioner-in-Streaming-with-parallelism-1-adding-latency-td13766.html

In some link here :
https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks

they have mentioned that the

 PipelinedSubpartition is a pipelined implementation to support streaming
data exchange. The SpillableSubpartition is a blocking implementation to
support batch data exchange.

I am not sure how would i use these or reduce latency from map ->
partitioner -> flatmap .






--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Partitioner-is-spending-around-2-to-4-minutes-while-pushing-data-to-next-operator-tp13913.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Mime
View raw message