flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gyula Fóra <gyf...@apache.org>
Subject Re: Turn lazy operator execution off for streaming jobs
Date Fri, 23 Jan 2015 13:14:30 GMT
Hey,

I checked it with our examples, it seems to working properly and exactly
how we wanted :).

One minor thing, we have a jira open for this so if you could put that in
the commit message that would be good.
https://issues.apache.org/jira/browse/FLINK-1425

Thanks!
Gyula


On Fri, Jan 23, 2015 at 11:47 AM, Gyula Fóra <gyfora@apache.org> wrote:

> Great, thanks! It was fast :)
>
> I will try to check it out in the afternoon!
>
> Gyula
>
> On Fri, Jan 23, 2015 at 11:37 AM, Ufuk Celebi <uce@apache.org> wrote:
>
>> On 22 Jan 2015, at 18:10, Stephan Ewen <sewen@apache.org> wrote:
>>
>> > So the crux is that the JobManager has a location for the sender task
>> (and
>> > tell that to the receivers) before the senders have registered their
>> > transfer queues.
>> >
>> > Can we just establish a "happens-before" there?
>> > - The TaskManager may send the "ack" to the deployment call only after
>> all
>> > queues are registered (might even be like this now)
>> > - The job manager updates receivers only with locations of senders that
>> > have switched to "running", not with ones that are in "deploying.
>> >
>> > Would that fix it?
>>
>> I've just looked into it and it turns out that I had already implemented
>> this in the current master. As long as the producer has not changed to
>> RUNNING, the consumer will be scheduled with an UNKNOWN input channel. The
>> problem that Till mentioned was specific to the test case where it was hard
>> coded to a local input channel.
>>
>> @Gyula: I've implemented the schedule mode as suggested by Stephan here:
>> https://github.com/uce/incubator-flink/tree/schedule-all
>>
>> You can set the schedule mode with
>>
>> jobGraph.setScheduleMode(ScheduleMode.ALL)
>>
>> I have just tested it with a small test case in JobMangerITCase. If you
>> have time, you could try it out with some of the streaming programs. If
>> this is what you need, I can open a PR later. :)
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message