apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (APEXCORE-146) Add A Clone Partitioner Which Sends The Same Data To Each Partition
Date Thu, 26 May 2016 19:43:12 GMT

    [ https://issues.apache.org/jira/browse/APEXCORE-146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302783#comment-15302783

ASF GitHub Bot commented on APEXCORE-146:

GitHub user ilganeli opened a pull request:


    [APEXCORE-146] Add ClonePartitioner

    **Note:** This is still marked as an Apex issue even though it should live in Malhar.
Once the JIRA is updated I'll update title and reference. 
    * Created a Clone partitioner similar to the StatelessPartitioner that assigns all data
to all partitions
    * Added a simple unit test suite to test scale up and scale down
    Is there a good place/way to add a test to verify that data is partitioned appropriately?
For example, the stateless partitioner assigns keys based on the nearest power of two - is
that validated anywhere?
    How is the serialVersionUUID generated?
    Does the manual assignment of ports to PartitionKeys need to happen for each Partition?
There is a comment in the Partition class that by default all data is sent to all partitions.
However, digging into the implementation in depth doesn't show that behavior. Instead, it
seems data is assigned based on the mask associated with each Partition. 
    This references: https://issues.apache.org/jira/browse/APEXCORE-146

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/ilganeli/incubator-apex-malhar APEXCORE-146

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #297
commit 74f6fd5113a3d488015a9817790018dd63b5e8cb
Author: Ilya Ganelin <ilya.ganelin@capitalone.com>
Date:   2016-05-26T19:39:24Z

    Added a new partitioner that replicates data across all partitions by default.


> Add A Clone Partitioner Which Sends The Same Data To Each Partition
> -------------------------------------------------------------------
>                 Key: APEXCORE-146
>                 URL: https://issues.apache.org/jira/browse/APEXCORE-146
>             Project: Apache Apex Core
>          Issue Type: New Feature
>            Reporter: Timothy Farkas
>            Assignee: Ilya Ganelin
> This should go into com.datatorrent.common.partitioner and would be very similar to the

This message was sent by Atlassian JIRA

View raw message