Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
User-Agent: K-9 Mail for Android
In-Reply-To: 
 <CAAdrtT1UzvDd-8tHy1nyWV=rjf7HkOgwJGOo1Rixk1nFR-n=mg@mail.gmail.com>
References: <56AF37CD.6060204@mailbox.tu-berlin.de>
 <CAAdrtT1UzvDd-8tHy1nyWV=rjf7HkOgwJGOo1Rixk1nFR-n=mg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----LW5M0LIHKOB59OUYYLPXU6GG3PLNQJ"
Content-Transfer-Encoding: 8bit
Subject: Re: DataExchangeMode.BATCH in iterations
From: Fridtjof Sander <fsander@mailbox.tu-berlin.de>
Date: Mon, 1 Feb 2016 13:04:35 +0100
To: <dev@flink.apache.org>
Message-ID: <E3958EA3-80DC-4365-9425-1AB4B3351230@mailbox.tu-berlin.de>

------LW5M0LIHKOB59OUYYLPXU6GG3PLNQJ
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="UTF-8"

Hi Fabian, 

thanks for your explanation!

Yeah, I figured that if an easy fix exists, you would have done that yourself. This is more for me to understand the conceptual problem.

But back to the pipeline-requirement: Doesn't zipWithIndex violate that too, then? It's also a mapPartitions, collect + broadcast, plus another mapPartitions. This should roughly be the same procedure as building a histogram and propagate partition boundaries, right?. Not much going on there with pipelining. However, I hadn't problems with zipWithIndex inside iterations.

Also, is there a difference between the "materialization" you mentioned and the execution of a datasink operator?

Again, if all that is written somewhere, just throw me the link, I don't want to waste your time.

Best
Fridtjof

Am 1. Februar 2016 12:21:21 MEZ, schrieb Fabian Hueske <fhueske@gmail.com>:
>Hi Fridtjof,
>
>the range partitioner works by building a histogram for the
>partitioning
>key. This requires a pass over the whole intermediate data set which
>means
>it needs to be materialized and cannot be processed in a pipelined
>fashion.
>However, pipelined data exchange strategies are a requirement for the
>data
>flows which are executed for iteration bodies.
>
>This is nothing that can be easily fixed at the moment. Touching this
>part
>of the runtime code would have major implications.
>I afraid, but I believe we have to accept this restriction.
>
>Best, Fabian
>
>
>2016-02-01 11:47 GMT+01:00 Fridtjof Sander
><fsander@mailbox.tu-berlin.de>:
>
>> Dear Flink-Devs,
>>
>> I recently ran into a problem where range-partitioning within
>iterations
>> would be useful.
>>
>> In the PR for range-partitioning it is said, this doesn't work
>because of
>> some batched data-exchange mode.
>> https://github.com/apache/flink/pull/1255
>>
>> I would like to understand the issue with that, but could not find
>> articles/blog posts/etc to read about that.
>>
>> Do you have some pointers for me? Code will also work if the concept
>gets
>> clear from it.
>>
>> Thanks for your time!
>>
>> Best, Fridtjof
>>

------LW5M0LIHKOB59OUYYLPXU6GG3PLNQJ--