Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Content-Type: text/plain; charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2104\))
Subject: Re: Scheduling in BATCH execution mode?
From: Ufuk Celebi <uce@apache.org>
In-Reply-To: <55F0591A.7070004@inet.tu-berlin.de>
Date: Wed, 9 Sep 2015 18:11:53 +0200
Content-Transfer-Encoding: quoted-printable
Message-Id: <10D22805-447A-4D45-A042-2BFC744E01EE@apache.org>
References: <55F0591A.7070004@inet.tu-berlin.de>
To: dev@flink.apache.org

Hey Niklas,

this is very much hidden unfortunately. You can find it in =
Execution#markFinished.

The last partition to be finished triggers the scheduling of the =
receivers.

=46rom your comments I see that you have dug through the network stack =
code quite a bit. If you are interested, we can have a chat about =
refactoring things like the scheduling of the receivers to be more =
accessible/transparent.

=E2=80=93 Ufuk

> On 09 Sep 2015, at 18:06, Niklas Semmler <niklas@inet.tu-berlin.de> =
wrote:
>=20
> Hello Flink community,
>=20
> what is the equivalent of the ScheduleOrUpdateConsumers message in the =
pipeline execution mode for the batch execution mode?
>=20
> When I run a WordCount in pipeline mode, the scheduling of the =
receiving tasks is initiated in the ResultPartition class via the =
function notifyPipelinedConsumers*. This leads to a  =
ScheduleOrUpdateConsumers message being sent to the JobManager and the =
JobManager takes care of the rest.
>=20
> In the batch mode this does not seem to be the case, as the =
notifyPipelinedConsumers function will only work in the pipeline =
execution mode:
>=20
> > if (partitionType.isPipelined() && !hasNotifiedPipelinedConsumers)
>=20
> So then, how is the consumer scheduled, or at least notified of the =
consumable partition?
>=20
> Cheers,
> Niklas
>=20
> * =
https://github.com/apache/flink/blob/572a45b379ca2231d772db4f115749fa08afc=
d10/flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partit=
ion/ResultPartition.java#L416