flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ufuk Celebi <u.cel...@fu-berlin.de>
Subject Re: Iterative stream processing - cyclic JobGraph?
Date Wed, 09 Jul 2014 08:26:50 GMT
Hey Gyula,

the BlockingBackChannel is only used for data feedback. The synchronization happens with a
separate barrier called SuperstepBarrier. (For delta iterations there is also a further SolutionSetUpdateBarrier).
But this are more "runtime hacks" (the DAG still looks like a DAG to the execution engine)
and you would have to think about how to model this stuff for streams.

Best,

Ufuk

On 09 Jul 2014, at 10:14, Gyula Fóra <gyula.fora@gmail.com> wrote:

> Hey,
> 
> So we started digging through the IterativeDataSet and it seems like you
> are using the BlockingBackChannel as this special feedback edge. Does it
> work as a synchronization barrier would in BSP? If so it is probably not
> suitable for processing continuous dataflows is it?
> 
> Gyula
> 
> 
> 
> 
> On Tue, Jul 8, 2014 at 4:33 PM, Stephan Ewen <sewen@apache.org> wrote:
> 
>> Cyclic graphs are a problem for both the scheduler and the
>> currently-under-works recovery logic.
>> 
>> General cyclic graphs are a bit of a design problem.
>> 
>> I think what we need in the future is something like a special feedback
>> edge that closes the cycle and that is synchronized (in batches /
>> generations) to make it transparent to the recovery and scheduling logic.
>> 
>> Stephan
>> 


Mime
View raw message