flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Union/append performance question
Date Mon, 07 Sep 2015 13:15:39 GMT
Union, like all operators, is lazy. When you call union, it only builds a
"union stream", that unions when you execute the task. So nothing is added
before you call "env.execute()"

After you call "env.execute()" and then union again, you will re-execute
the entire history of computation to compute the data set that you union
with. Hence, for incremental computations, union() is probably not a good
choice, unless you persist intermediate data (seamless support for that is
WIP).

Stephan


On Mon, Sep 7, 2015 at 2:56 PM, Flavio Pompermaier <pompermaier@okkam.it>
wrote:

> Hi to all,
> I have a job where I have to incrementally add Tuples to a dataset (in a
> while loop).
> Is union() the best operator for this task or is there a more performant
> operator for this task?
> Does union affect the read of already existing elements or it just appends
> the new ones somewhere?
>
> Best,
> Flavio
>
>
>

Mime
View raw message