flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niklas Semmler <nsemm...@inet.tu-berlin.de>
Subject Re: Scheduling in BATCH execution mode?
Date Wed, 09 Sep 2015 17:31:23 GMT
Hello Ufuk,

thanks for you amazingly quick reply.

I have seen the markFinished in Execution.java, but if I get it right, 
this is simply used to stop a task. The ScheduleOrUpdateConsumers 
message in the pipeline case on the other hand is notifying the 
consumers that a pipelined partition is ready and can now be consumed. 
Can you give me a hint on how the receiver is notified of a consumable 
partition in the batch case?

And yeah I would be great if we could have a chat :).

Best,
Niklas

On 09.09.2015 18:11, Ufuk Celebi wrote:
> Hey Niklas,
>
> this is very much hidden unfortunately. You can find it in Execution#markFinished.
>
> The last partition to be finished triggers the scheduling of the receivers.
>
>  From your comments I see that you have dug through the network stack code quite a bit.
If you are interested, we can have a chat about refactoring things like the scheduling of
the receivers to be more accessible/transparent.
>
> – Ufuk
>
>> On 09 Sep 2015, at 18:06, Niklas Semmler <niklas@inet.tu-berlin.de> wrote:
>>
>> Hello Flink community,
>>
>> what is the equivalent of the ScheduleOrUpdateConsumers message in the pipeline execution
mode for the batch execution mode?
>>
>> When I run a WordCount in pipeline mode, the scheduling of the receiving tasks is
initiated in the ResultPartition class via the function notifyPipelinedConsumers*. This leads
to a  ScheduleOrUpdateConsumers message being sent to the JobManager and the JobManager takes
care of the rest.
>>
>> In the batch mode this does not seem to be the case, as the notifyPipelinedConsumers
function will only work in the pipeline execution mode:
>>
>>> if (partitionType.isPipelined() && !hasNotifiedPipelinedConsumers)
>>
>> So then, how is the consumer scheduled, or at least notified of the consumable partition?
>>
>> Cheers,
>> Niklas
>>
>> * https://github.com/apache/flink/blob/572a45b379ca2231d772db4f115749fa08afcd10/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/ResultPartition.java#L416
>

Mime
View raw message