flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: [flink-streaming] Regarding loops in the Job Graph
Date Wed, 21 Jan 2015 18:36:27 GMT
Hi Paris!

The Streaming API allows you to define iterations, where parts of the
stream are fed back. Do those work for you?

In general, cyclic flows are a tricky thing, as the topological order of
operators is needed for scheduling (may not be important for continuous
streams) but also for a clear producer/consumer relationship, which is
important for fault tolerance techniques.

Currently, the JobManager topologically sorts the job graph and starts
scheduling operators. I am surprised to hear that a graph with cyclic
dependencies works...


Stephan




Stephan


On Wed, Jan 21, 2015 at 2:57 AM, Paris Carbone <parisc@kth.se> wrote:

> Hello,
>
> While implementing the SAMOA adapter for Flink-Streaming we stumbled upon
> the need to allow loops (or circular dependencies) in the job graph. Many
> incremental machine learning tasks define loops already  and there is no
> trivial way of getting around it. In the streaming job graph builder there
> is only a check that does not allow the user to submit graphs with loops,
> however, from what Gyula told me, if the check is removed the streaming job
> runs as expected. Is there (still) a major reason for having this check, at
> least in the streaming component?
>
> Paris

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message