beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-434) Limit the number of output files a beam-examples execution writes
Date Thu, 14 Jul 2016 00:01:35 GMT

    [ https://issues.apache.org/jira/browse/BEAM-434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376035#comment-15376035
] 

ASF GitHub Bot commented on BEAM-434:
-------------------------------------

GitHub user tgroh opened a pull request:

    https://github.com/apache/incubator-beam/pull/651

    [BEAM-434] Control the number of output shards in the Direct Runner

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [ ] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`. (Even better, enable
           Travis-CI on your fork and ensure the whole test matrix passes).
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt).
    
    ---
    
    Add a Write Override Factory that limits the number of shards is
    unspecified. This ensures that we will not write an output file per-key
    due to bundling.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/tgroh/incubator-beam write_sharding

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/incubator-beam/pull/651.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #651
    
----
commit dfecd170d67c90ae20a4baed3986bd1c7116e79e
Author: Thomas Groh <tgroh@google.com>
Date:   2016-07-13T21:26:10Z

    Control the number of output shards in the Direct Runner
    
    Add a Write Override Factory that limits the number of shards is
    unspecified. This ensures that we will not write an output file per-key
    due to bundling.

----


> Limit the number of output files a beam-examples execution writes
> -----------------------------------------------------------------
>
>                 Key: BEAM-434
>                 URL: https://issues.apache.org/jira/browse/BEAM-434
>             Project: Beam
>          Issue Type: Bug
>          Components: examples-java
>            Reporter: Amit Sela
>            Assignee: Amit Sela
>            Priority: Minor
>
> When using `TextIO.Write.to("/path/to/output")` without any restrictions on the number
of shards, it might generate many output files (depending on your input), for WordCount for
example, you'll get as many output files as unique words in your input.
> Since I think examples are expected to execute in a friendly manner to "see" what it
does and not optimize for performance in some way, I suggest to use `withoutSharding()` when
writing the example output to an output file.
> Examples I could find that behave this way:
> org.apache.beam.examples.WordCount
> org.apache.beam.examples.complete.TfIdf
> org.apache.beam.examples.cookbook.DeDupExample



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message