Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@beam.incubator.apache.org
Date: Wed, 5 Oct 2016 18:08:20 +0000 (UTC)
From: "ASF GitHub Bot (JIRA)" <jira@apache.org>
To: commits@beam.incubator.apache.org
Message-ID: <JIRA.12965418.1462484467000.745329.1475690900629@Atlassian.JIRA>
In-Reply-To: <JIRA.12965418.1462484467000@Atlassian.JIRA>
References: <JIRA.12965418.1462484467000@Atlassian.JIRA> <JIRA.12965418.1462484467907@arcas>
Subject: [jira] [Commented] (BEAM-259) Execute selected RunnableOnService
 tests with Spark runner
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 05 Oct 2016 18:08:26 -0000


    [ https://issues.apache.org/jira/browse/BEAM-259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15549524#comment-15549524 ] 

ASF GitHub Bot commented on BEAM-259:
-------------------------------------

GitHub user amitsela opened a pull request:

    https://github.com/apache/incubator-beam/pull/1055

    [BEAM-259] Enable RunnableOnService for batch.

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [ ] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
    
    ---
    
    Handle empty Flatten for bounded.
    
    Spark will bubble out a SparkException for user code failure, so this won't catch. Asserting on the
    error message should be good enough.
    
    outputWithTimestamp should handle start/finishBundle as well.
    
    Explode WindowedValues before processing.
    
    sideOutputWithTimestamp to address start/finishBundle.
    
    SideInput with windows.
    
    Unused for now, remove.
    
    Take sideInput window startegy into account, for combine as well.
    
    reduce code duplication.
    
    Spark combine support.
    
    reuse code where possible.
    
    Expose sideInputs and insertDefault in Combine.Globally for direct translation.
    
    Direct translation of Combine.Globally into Spark's aggregate function.
    
    Make default run with 4 cores by default - makes tests run with multiple threads, but not too many.
    
    SideInputReader for the Spark runner.
    
    A common abstraction for Keyed and Global implementation.
    
    Implement Combine.Globally via Spark's aggregate.
    
    runnable-on-service profile doesn't need pluginManagement.
    
    Removing test as it does not follow a deterministic combine implementation.
    
    Context reuse is mostly for testing. To avoid a test failure that will stop the context and fail all
    following tests we need to recreate the context if it's stopped as well.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/amitsela/incubator-beam BEAM-259

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-beam/pull/1055.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1055
    
----
commit 53ed9951203da7f7e2cff0ef4d9ec64f37a0f29d
Author: Sela <ansela@paypal.com>
Date:   2016-09-23T10:32:28Z

    Enable RunnableOnService for batch.
    
    Handle empty Flatten for bounded.
    
    Spark will bubble out a SparkException for user code failure, so this won't catch. Asserting on the
    error message should be good enough.
    
    outputWithTimestamp should handle start/finishBundle as well.
    
    Explode WindowedValues before processing.
    
    sideOutputWithTimestamp to address start/finishBundle.
    
    SideInput with windows.
    
    Unused for now, remove.
    
    Take sideInput window startegy into account, for combine as well.
    
    reduce code duplication.
    
    Spark combine support.
    
    reuse code where possible.
    
    Expose sideInputs and insertDefault in Combine.Globally for direct translation.
    
    Direct translation of Combine.Globally into Spark's aggregate function.
    
    Make default run with 4 cores by default - makes tests run with multiple threads, but not too many.
    
    SideInputReader for the Spark runner.
    
    A common abstraction for Keyed and Global implementation.
    
    Implement Combine.Globally via Spark's aggregate.
    
    runnable-on-service profile doesn't need pluginManagement.
    
    Removing test as it does not follow a deterministic combine implementation.
    
    Context reuse is mostly for testing. To avoid a test failure that will stop the context and fail all
    following tests we need to recreate the context if it's stopped as well.

----


> Execute selected RunnableOnService tests with Spark runner
> ----------------------------------------------------------
>
>                 Key: BEAM-259
>                 URL: https://issues.apache.org/jira/browse/BEAM-259
>             Project: Beam
>          Issue Type: Test
>          Components: runner-spark
>            Reporter: Kenneth Knowles
>            Assignee: Amit Sela
>


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)