flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <...@apache.org>
Subject Re: Turn lazy operator execution off for streaming jobs
Date Fri, 23 Jan 2015 10:37:19 GMT
On 22 Jan 2015, at 18:10, Stephan Ewen <sewen@apache.org> wrote:

> So the crux is that the JobManager has a location for the sender task (and
> tell that to the receivers) before the senders have registered their
> transfer queues.
> 
> Can we just establish a "happens-before" there?
> - The TaskManager may send the "ack" to the deployment call only after all
> queues are registered (might even be like this now)
> - The job manager updates receivers only with locations of senders that
> have switched to "running", not with ones that are in "deploying.
> 
> Would that fix it?

I've just looked into it and it turns out that I had already implemented this in the current
master. As long as the producer has not changed to RUNNING, the consumer will be scheduled
with an UNKNOWN input channel. The problem that Till mentioned was specific to the test case
where it was hard coded to a local input channel.

@Gyula: I've implemented the schedule mode as suggested by Stephan here: https://github.com/uce/incubator-flink/tree/schedule-all

You can set the schedule mode with

jobGraph.setScheduleMode(ScheduleMode.ALL)

I have just tested it with a small test case in JobMangerITCase. If you have time, you could
try it out with some of the streaming programs. If this is what you need, I can open a PR
later. :)
Mime
View raw message