flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephan Ewen <se...@apache.org>
Subject Re: Flink Iterations Ordering
Date Tue, 03 May 2016 12:50:46 GMT
Hi!

The order in which the elements arrive in an iteration HEAD is the order in
which the last operator in the loop (the TAIL) produces them. If that is a
deterministic ordering (because of a sorted reduce, for example), then you
should be able to rely on the order.

Otherwise, the order of elements can always change every time elements are
re-distributed across tasks (for example during a key-based shuffle, a
broadcast, or a change of parallelism).

Greetings,
Stephan


On Mon, May 2, 2016 at 2:50 PM, Aljoscha Krettek <aljoscha@apache.org>
wrote:

> Hi,
> as I understand it the order of elements will not be preserved across
> iteration supersets. But maybe some-one else knows more.
>
> Cheers,
> Aljoscha
>
> On Thu, 28 Apr 2016 at 00:23 David Kim <david.kim@braintreepayments.com>
> wrote:
>
>> Hello all,
>>
>> I read the documentation at [1] on iterations and had a question on
>> whether an assumption is safe to make.
>>
>> As partial solutions are continuously looping through the step function,
>> when new elements are added as iteration inputs will the insertion order of
>> all of the elements be preserved?
>>
>> Example:
>>
>> Current partial solutions going through step function are: A, B, C.
>>
>> At a later time we add (in order) elements D, E.
>>
>> Will the iteration result always be A,B,C,D,E?
>>
>> References:
>>
>> [1] *https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html
>> <https://ci.apache.org/projects/flink/flink-docs-master/apis/batch/iterations.html>*
>>
>

Mime
View raw message