Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
References: <76a498af-2503-9a70-d511-c7906ebb1d24@rasumi.net>
In-Reply-To: <76a498af-2503-9a70-d511-c7906ebb1d24@rasumi.net>
From: Aljoscha Krettek <aljoscha@apache.org>
Date: Mon, 27 Jun 2016 09:24:19 +0000
Message-ID: <CANMXwW3vMvCMR7_AKh591ojXGd7t4ba=7Hv-8336rpYqm2G5uA@mail.gmail.com>
Subject: Re: forward()
To: Dev <dev@flink.apache.org>
Content-Type: multipart/alternative; boundary=001a114a6e9461205005363f162c
archived-at: Mon, 27 Jun 2016 09:24:33 -0000

--001a114a6e9461205005363f162c
Content-Type: text/plain; charset=UTF-8

Hi,
you are right, this seems a bit strange. The reason why this works is that
selectChannels of ForwardPartitioner is never called.

In StreamingJobGraphGenerator.connect() operators are connected by
"physical" edges. If the ForwardPartitioner is set on a stream this will
choose the POINTWISE connection pattern, which does local forwarding from
partition 1 to 1, 2 to 2, and so on. Only if another partitioner is set
will the ALL_TO_ALL pattern be used. In that case selectChannels() of the
partitioner is actually called for decision where to send to.

Cheers,
Aljoscha

On Mon, 27 Jun 2016 at 01:57 Marius Melzer <marius@rasumi.net> wrote:

> Hi,
>
> I was reading a bit of code about partitioning and I have two questions
> about the forward() method of DataStream:
>
> 1) I was before under the impression that forwarding means sending
> records between operators with same parallelism always from partition 1
> to 1, 2 to 2 etc. But this doesn't seem the case because
> ForwardPartitioner#selectChannels returns new int[] {0} - which is
> interestingly the exact same code like GlobalPartitioner.
> 2) Why is there a forward()-statement anyways? Isn't this the default
> that messages are forwarded between same partitions or am I getting
> something completely wrong here? If so, how does it work and what would
> be a good use case for the forward() statement?
>
> Thanks,
> Marius
>

--001a114a6e9461205005363f162c--