flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Márton Balassi <balassi.mar...@gmail.com>
Subject Re: [streaming] Regarding loops in the job graph
Date Wed, 21 Jan 2015 11:15:49 GMT
As far as I remember when we were implementing the iteration for streaming
we deliberately "went off" the jobgraph and implemented the backward edge
as an in-memory hook (practically the same in batch but not blocking in the
backward channel).

When we asked the reasoning here on the dev list we got some reasoning for
keeping the check for the DAG property of the Jobgraph, but I could not
find the thread now.

On Wed, Jan 21, 2015 at 11:57 AM, Paris Carbone <parisc@kth.se> wrote:

> Hello,
> While implementing the SAMOA adapter for Flink-Streaming we stumbled upon
> the need to allow loops (or circular dependencies) in the job graph. Many
> incremental machine learning tasks define loops already  and there is no
> trivial way of getting around it. In the streaming job graph builder there
> is only a check that does not allow the user to submit graphs with loops,
> however, from what Gyula told me, if the check is removed the streaming job
> runs as expected. Is there (still) a major reason for having this check, at
> least in the streaming component?
> Paris

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message