flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Flink Iterations Ordering
Date Tue, 03 May 2016 12:50:46 GMT

The order in which the elements arrive in an iteration HEAD is the order in
which the last operator in the loop (the TAIL) produces them. If that is a
deterministic ordering (because of a sorted reduce, for example), then you
should be able to rely on the order.

Otherwise, the order of elements can always change every time elements are
re-distributed across tasks (for example during a key-based shuffle, a
broadcast, or a change of parallelism).


On Mon, May 2, 2016 at 2:50 PM, Aljoscha Krettek <aljoscha@apache.org>

> Hi,
> as I understand it the order of elements will not be preserved across
> iteration supersets. But maybe some-one else knows more.
> Cheers,
> Aljoscha
> On Thu, 28 Apr 2016 at 00:23 David Kim <david.kim@braintreepayments.com>
> wrote:
>> Hello all,
>> I read the documentation at [1] on iterations and had a question on
>> whether an assumption is safe to make.
>> As partial solutions are continuously looping through the step function,
>> when new elements are added as iteration inputs will the insertion order of
>> all of the elements be preserved?
>> Example:
>> Current partial solutions going through step function are: A, B, C.
>> At a later time we add (in order) elements D, E.
>> Will the iteration result always be A,B,C,D,E?
>> References:
>> [1] *https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html
>> <https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html>*

View raw message