flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Scheduling in BATCH execution mode?
Date Wed, 09 Sep 2015 16:11:53 GMT
Hey Niklas,

this is very much hidden unfortunately. You can find it in Execution#markFinished.

The last partition to be finished triggers the scheduling of the receivers.

From your comments I see that you have dug through the network stack code quite a bit. If
you are interested, we can have a chat about refactoring things like the scheduling of the
receivers to be more accessible/transparent.

– Ufuk

> On 09 Sep 2015, at 18:06, Niklas Semmler <niklas@inet.tu-berlin.de> wrote:
> Hello Flink community,
> what is the equivalent of the ScheduleOrUpdateConsumers message in the pipeline execution
mode for the batch execution mode?
> When I run a WordCount in pipeline mode, the scheduling of the receiving tasks is initiated
in the ResultPartition class via the function notifyPipelinedConsumers*. This leads to a 
ScheduleOrUpdateConsumers message being sent to the JobManager and the JobManager takes care
of the rest.
> In the batch mode this does not seem to be the case, as the notifyPipelinedConsumers
function will only work in the pipeline execution mode:
> > if (partitionType.isPipelined() && !hasNotifiedPipelinedConsumers)
> So then, how is the consumer scheduled, or at least notified of the consumable partition?
> Cheers,
> Niklas
> * https://github.com/apache/flink/blob/572a45b379ca2231d772db4f115749fa08afcd10/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartition.java#L416

View raw message