flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gábor Gévay <gga...@gmail.com>
Subject Re: Nested iterations
Date Thu, 01 Sep 2016 16:28:29 GMT
Hello Supun,

Unfortunately, nesting of Flink's iteration constructs are not
supported at the moment.

There are some workarounds though:

1. You can start a Flink job for each step of the iteration. Starting
a Flink job has some overhead, so this only works if there is a
sufficient amount of work in each iteration step. Moreover, this has
the disadvantage that the intermediate results are always need to be
written out and then read back between steps, which might have a
considerable performance impact.

2. If you have just a small fixed number of steps, then you can have a
for loop that "unrolls" all the iteration steps, and creates one large
Flink job. The code will be somewhat similar to the first approach,
but you don't call execute between the steps, and you don't write
intermediate results to a sink, but just use the DataSet from the
previous step. The disadvantage of this is that you might end up with
a too large Flink job, which might also hurt performance.


2016-09-01 18:09 GMT+02:00 Supun Kamburugamuve <supun06@gmail.com>:
> Hi,
> Does Flink support nested iterations? We are trying to develop a complex
> machine learning algorithm which has 3 iterations nested.
> Best,
> Supun..

View raw message