beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (BEAM-79) Gearpump runner
Date Thu, 18 May 2017 03:12:05 GMT

    [ https://issues.apache.org/jira/browse/BEAM-79?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16015131#comment-16015131
] 

ASF GitHub Bot commented on BEAM-79:
------------------------------------

GitHub user manuzhang opened a pull request:

    https://github.com/apache/beam/pull/3172

    [BEAM-79] Sync gearpump-runner with master

    Be sure to do all of the following to help us incorporate your contribution
    quickly and easily:
    
     - [ ] Make sure the PR title is formatted like:
       `[BEAM-<Jira issue #>] Description of pull request`
     - [ ] Make sure tests pass via `mvn clean verify`.
     - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue
           number, if there is one.
     - [ ] If this contribution is large, please file an Apache
           [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
    
    ---


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/manuzhang/beam sync-master

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/beam/pull/3172.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3172
    
----
commit 44c9385ddbcfbdf904c5d054754dca3ddde88ec2
Author: Dan Halperin <dhalperi@google.com>
Date:   2017-05-06T00:16:34Z

    [BEAM-2190] pom.xml: do a better job of dependency management
    
    Even if Beam appears to have the correct dependencies, we cannot
    guarantee that modules that depend on us transitively get the right
    dependencies. For example, even though grpc-protobuf-lite has
    protobuf-lite excluded, and the Maven Enforcer banned-dependencies
    check passes... if a user happens to get a transitive dependency on
    grpc-all first, they may pull in grpc-protobuf from that other source
    without the exclusion. Thus we need to exclude protobuf-lite from
    grpc-all as well.
    
    While we're here, also add guava-jdk5 to the set of banned dependencies,
    though (as above) we cannot currently properly identify the places it
    might be transitively exposed in a users' pom.xml.

commit 2b0e699b8f6a0aac26ac57c536e8d45cee1c3d2e
Author: Dan Halperin <dhalperi@google.com>
Date:   2017-05-08T18:35:56Z

    This closes #2934

commit 5bac40efcc7bcd9d53183579ec6089c23aeda782
Author: Dan Halperin <dhalperi@google.com>
Date:   2017-05-06T02:06:03Z

    [BEAM-2212] FileBasedSource: improve message when logging.
    
    ValueProvider should not be printed, rather the string instead.

commit 9423babd8f827e843723c218441e9a91aaa7b361
Author: Dan Halperin <dhalperi@google.com>
Date:   2017-05-08T16:59:16Z

    [BEAM-2212] FileBasedSource: refactor to remove uses of fileOrPatternSpec.get()
    
    Makes it less likely to have errors from printing ValueProviders instead of runtime values

commit e1791c3f83a007013625b43f9976e7ffa64e8a49
Author: Dan Halperin <dhalperi@google.com>
Date:   2017-05-08T18:48:42Z

    This closes #2956

commit 13e51c9cd114eec73c47b71f46214dd92ee81048
Author: Dan Halperin <dhalperi@google.com>
Date:   2017-05-06T02:49:29Z

    Convert Executor Services to use Daemon Threads
    
    This will cause the DirectRunner to automatically shut down when the
    worker threads are shut down.

commit 63c6bea3bbdb52964ac54073326d4ad99ba7432c
Author: Thomas Groh <tgroh@google.com>
Date:   2017-05-08T19:52:09Z

    This closes #2954

commit 50a4c56de1d3aa0e51e8b2b08b71715be4c2ef80
Author: Sourabh Bajaj <sourabhbajaj@google.com>
Date:   2017-05-07T20:55:20Z

    [BEAM-2206] Move pipelineOptions into options modules

commit 604c2f59e2a928a95106814b0b55b8097dc39faa
Author: Ahmet Altay <altay@google.com>
Date:   2017-05-08T19:56:15Z

    This closes #2925

commit d3fc7d18a4a3c394d32d85ef34c08a85712a8b18
Author: Reuven Lax <relax@google.com>
Date:   2017-05-08T16:06:55Z

    TableDescription is allowed to be null.

commit c481396edc7a3e906cd79dcc2a356baf9c93895e
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-05-08T20:41:01Z

    This closes #2953

commit 1fd15bd7441e1be72feb3f9535ffe536d730a531
Author: Reuven Lax <relax@google.com>
Date:   2017-05-07T18:50:08Z

    Shortens PubsubIO names for reading/writing messages.

commit d1914ed0c3195e1de2f707e2bb7bc57597b834fe
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-05-08T20:45:25Z

    This closes #2950

commit b7635aadc2532b5c3bdc38f070865091ebf901c1
Author: Ahmet Altay <altay@google.com>
Date:   2017-05-08T20:42:02Z

    Remove apex README.md. This information is already in the website in
    quickstart and apex runner pages.
    https://github.com/apache/beam-site/pull/232 moves small bits of missing
    content.

commit 962e8796d68f9a81b0ae3d1b30d7a4bcadbdaf60
Author: Ahmet Altay <altay@google.com>
Date:   2017-05-08T21:28:36Z

    This closes #2962

commit fba4c010ce90db8b5cdac489d47126ca3ee8db65
Author: Ahmet Altay <altay@google.com>
Date:   2017-05-08T22:00:33Z

    Remove readme.md files that are covered in web site

commit 152c5bcc7f6fb53565ed132653384f4244542b59
Author: Ahmet Altay <altay@google.com>
Date:   2017-05-08T22:04:46Z

    This closes #2966

commit 78c29238aa471138fb55681d8700b095a3606e0d
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-05-08T21:54:17Z

    [BEAM-2224] Fix temp file management on Windows.

commit f315cc9ff649ccee9c9aff18f4af30cfbd58e5e5
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-05-08T23:49:05Z

    Closes #2965

commit fd1dda09fc4db48cfbc21afd6310320533fffe78
Author: Reuven Lax <relax@google.com>
Date:   2017-04-29T14:33:54Z

    [BEAM-2154] More scalable dynamic BigQueryIO.Write
    
    If too many tables are generated in a bundle, spill and group the results
    before writing files. Generating hundreds or thousands of file write buffers
    in a single bundle was causing workers to crash with out of memory.

commit 6a2586a4ee279f66fd4eae816e93fa8e397acf6b
Author: Eugene Kirpichov <kirpichov@google.com>
Date:   2017-05-09T01:02:14Z

    This closes #2883

commit 367fcb28d544934797d25cb34d54136b2d7d6e99
Author: Vikas Kedigehalli <vikasrk@google.com>
Date:   2017-05-06T02:24:51Z

    Remove google api BackOff usage from sdks/core

commit 87578a6a69f15bca502d9b4c15999aa383e539a5
Author: Vikas Kedigehalli <vikasrk@google.com>
Date:   2017-05-06T02:33:45Z

    Use guava Base64 encoding instead of google api client

commit 88e044fda0a7d0ea1372f075a4c7a4bd54af47da
Author: Luke Cwik <lcwik@google.com>
Date:   2017-05-09T02:20:01Z

    [BEAM-1871] Remove Google API client dependency from sdks/core
    
    This closes #2937

commit 16f355f7e481ebf029c0edb878742f6fea57b6cd
Author: Dan Halperin <dhalperi@google.com>
Date:   2017-05-08T22:37:29Z

    [BEAM-2211] DataflowRunner: remove validation of file read/write paths
    
    Now that users can implement and register custom FileSystems,
    we can no longer really effectively validate filesystems they
    can read or write files from. They can even register file://
    to point to some HDFS path, e.g.,

commit 23731fe7ab12ea5d3ebb7b3f16c788b83d0caf0b
Author: Dan Halperin <dhalperi@google.com>
Date:   2017-05-09T03:13:38Z

    This closes #2968

commit 27e9a060ed593b2b53b88481591f14b1a274c61b
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-05-05T23:20:37Z

    Remove explicit used of nested contexts.
    
        find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\),  *context.nested..[)]/\1)/'
        find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\),  *nestedContext[)]/\1)/'
        find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\),  *Context.NESTED[)]/\1)/'
        find . -type f -name '*.java' | xargs sed -i '' 's/\([.]..code[(].*\),  *[^ ]*.Context.NESTED[)]/\1)/'
    
    Added back explicit context in CoGbkResult.java due to compile error.

commit 996dce37b76b103f104328b7caa65f73a1bcb15a
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-05-05T23:36:47Z

    Remove contexts from coders where they'll never be used.

commit b7f3341ed4fc073a834a55773bcb4a2c7f821c52
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-05-06T00:24:02Z

    automated context removal or redirection

commit 45e09b2df7a97cfbbe0d8013c16cfdd7a55afbde
Author: Robert Bradshaw <robertwb@gmail.com>
Date:   2017-05-06T00:27:13Z

    get it compiling

----


> Gearpump runner
> ---------------
>
>                 Key: BEAM-79
>                 URL: https://issues.apache.org/jira/browse/BEAM-79
>             Project: Beam
>          Issue Type: New Feature
>          Components: runner-gearpump
>            Reporter: Tyler Akidau
>            Assignee: Manu Zhang
>
> Intel is submitting Gearpump (http://www.gearpump.io) to ASF (https://wiki.apache.org/incubator/GearpumpProposal).
Appears to be a mix of low-level primitives a la MillWheel, with some higher level primitives
like non-merging windowing mixed in. Seems like it would make a nice Beam runner.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message