Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
MIME-Version: 1.0
In-Reply-To: 
 <CY1PR0601MB12282C11B3D9E3EEB4D53CF694610@CY1PR0601MB1228.namprd06.prod.outlook.com>
References: 
 <CY1PR0601MB12282C11B3D9E3EEB4D53CF694610@CY1PR0601MB1228.namprd06.prod.outlook.com>
Date: Mon, 25 Apr 2016 11:27:28 +0200
Message-ID: 
 <CAC27z=PgO-0G7CXA11v0MqeCsXqrv4NOKeP-rQ14=Y7-Khsp7w@mail.gmail.com>
Subject: Re: Partition problem
From: Till Rohrmann <trohrmann@apache.org>
To: dev@flink.apache.org
Content-Type: multipart/alternative; boundary=001a1130d2180933c905314bc9c9

--001a1130d2180933c905314bc9c9
Content-Type: text/plain; charset=UTF-8

Hi Andrew,

I think the problem is that you assume that both matrices have the same
partitioning. If you guarantee that this is the case, then you can use the
subtask index as the block index. But in the general case this is not true,
and then you have to calculate the blocks by first assigning a block index
(e.g. rows with 0-9 index are assigned to block 0, rows with 10-19 assigned
to block 1, etc.) and then create the blocks by reducing on this block
index. That's because the distribution of the individual rows in the
cluster is not necessarily the same between two matrices.

Cheers,
Till

On Mon, Apr 25, 2016 at 1:40 AM, Andrew Palumbo <ap.dev@outlook.com> wrote:

> Hi All,
>
>
> I've run into a problem with empty partitions when the number of elements
> in a DataSet is less than the Degree of Parallelism.  I've created a gist
> here to describe it:
>
>
> https://gist.github.com/andrewpalumbo/1768dac6d2f5fdf963abacabd859aaf3
>
>
> I have two 2x2 matrices, Matrix A and Matrix B and an execution
> environment where the degree of parallelism is 4. Both matrices   are
> blockified in  2 different DataSet s . In this case (the case of a 2x2
> matrices with 4 partitions) this means that each row goes into a partition
> leaving 2 empty partitions. In Matrix A, the rows go into partitions 0, 1.
> However the rows of Matrix B end up in partitions 1, 2. I assign the
> ordinal index of the blockified matrix's partition to its block, and then
> join on that index.
>
>
> However in this case, with differently partitioned matrices of the same
> geometry, the intersection of the blockified matrices' indices is 1, and
> partitions 0 and 2 are dropped.
>
>
> I've tried explicitly defining the dop for Matrix B using the count of
> non-empty partitions in Matrix A, however this changes the order of the
> DataSet, placing partition 2 into partition 0.
>
>
> Is there a way to make sure that these datasets are partitioned in the
> same way?
>
>
> Thank you,
>
>
> Andy
>
>
>

--001a1130d2180933c905314bc9c9--