flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chesnay Schepler <ches...@apache.org>
Subject Re: Partitioner is spending around 2 to 4 minutes while pushing data to next operator
Date Thu, 22 Jun 2017 12:18:22 GMT
So let's get the obvious question out of the way:

Why are you adding a partitioner when your parallelism is 1?

On 22.06.2017 11:58, sohimankotia wrote:
> I have a execution flow (Streaming Job) with parallelism 1.
>
>   source -> map -> partitioner -> flatmap -> sink
>
> Since adding partitioner will start new thread but partitioner is spending
> average of 2 to 4 minutes while moving data from map to flat map .
>
> For more details about this  :
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Using-Custom-Partitioner-in-Streaming-with-parallelism-1-adding-latency-td13766.html
>
> In some link here :
> https://cwiki.apache.org/confluence/display/FLINK/Data+exchange+between+tasks
>
> they have mentioned that the
>
>   PipelinedSubpartition is a pipelined implementation to support streaming
> data exchange. The SpillableSubpartition is a blocking implementation to
> support batch data exchange.
>
> I am not sure how would i use these or reduce latency from map ->
> partitioner -> flatmap .
>
>
>
>
>
>
> --
> View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Partitioner-is-spending-around-2-to-4-minutes-while-pushing-data-to-next-operator-tp13913.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.
>


Mime
View raw message