flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Supun Kamburugamuve <supu...@gmail.com>
Subject Re: Nested iterations
Date Thu, 01 Sep 2016 16:31:23 GMT
Thanks Gabor. I was thinking about starting separate jobs.

Is there any plans to support nested loops in the future?

Thanks,
Supun..

On Thu, Sep 1, 2016 at 12:28 PM, Gábor Gévay <ggab90@gmail.com> wrote:

> Hello Supun,
>
> Unfortunately, nesting of Flink's iteration constructs are not
> supported at the moment.
>
> There are some workarounds though:
>
> 1. You can start a Flink job for each step of the iteration. Starting
> a Flink job has some overhead, so this only works if there is a
> sufficient amount of work in each iteration step. Moreover, this has
> the disadvantage that the intermediate results are always need to be
> written out and then read back between steps, which might have a
> considerable performance impact.
>
> 2. If you have just a small fixed number of steps, then you can have a
> for loop that "unrolls" all the iteration steps, and creates one large
> Flink job. The code will be somewhat similar to the first approach,
> but you don't call execute between the steps, and you don't write
> intermediate results to a sink, but just use the DataSet from the
> previous step. The disadvantage of this is that you might end up with
> a too large Flink job, which might also hurt performance.
>
> Best,
> Gábor
>
>
>
>
>
>
> 2016-09-01 18:09 GMT+02:00 Supun Kamburugamuve <supun06@gmail.com>:
> > Hi,
> >
> > Does Flink support nested iterations? We are trying to develop a complex
> > machine learning algorithm which has 3 iterations nested.
> >
> > Best,
> > Supun..
> >
> >
>



-- 
Supun Kamburugamuve
Member, Apache Software Foundation; http://www.apache.org
E-mail: supun@apache.o <supun06@gmail.com>rg;  Mobile: +1 812 219 2563

Mime
View raw message