Return-Path: X-Original-To: apmail-beam-commits-archive@minotaur.apache.org Delivered-To: apmail-beam-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 2D68619399 for ; Fri, 8 Apr 2016 18:11:31 +0000 (UTC) Received: (qmail 39054 invoked by uid 500); 8 Apr 2016 18:11:31 -0000 Delivered-To: apmail-beam-commits-archive@beam.apache.org Received: (qmail 39005 invoked by uid 500); 8 Apr 2016 18:11:31 -0000 Mailing-List: contact commits-help@beam.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.incubator.apache.org Delivered-To: mailing list commits@beam.incubator.apache.org Received: (qmail 38996 invoked by uid 99); 8 Apr 2016 18:11:31 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2016 18:11:31 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 8C9E7C0D34 for ; Fri, 8 Apr 2016 18:11:30 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.021 X-Spam-Level: X-Spam-Status: No, score=-4.021 tagged_above=-999 required=6.31 tests=[KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id mBx9tx4_tnzH for ; Fri, 8 Apr 2016 18:11:28 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 413755FACD for ; Fri, 8 Apr 2016 18:11:27 +0000 (UTC) Received: (qmail 38544 invoked by uid 99); 8 Apr 2016 18:11:26 -0000 Received: from arcas.apache.org (HELO arcas) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 08 Apr 2016 18:11:26 +0000 Received: from arcas.apache.org (localhost [127.0.0.1]) by arcas (Postfix) with ESMTP id 0DE2D2C14F6 for ; Fri, 8 Apr 2016 18:11:26 +0000 (UTC) Date: Fri, 8 Apr 2016 18:11:26 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: commits@beam.incubator.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (BEAM-22) DirectPipelineRunner: support for unbounded collections MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/BEAM-22?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15232629#comment-15232629 ] ASF GitHub Bot commented on BEAM-22: ------------------------------------ GitHub user tgroh opened a pull request: https://github.com/apache/incubator-beam/pull/149 [BEAM-22] Add WindowIntoEvaluatorFactory Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [x] Make sure the PR title is formatted like: `[BEAM-] Description of pull request` - [x] Make sure tests pass via `mvn clean verify`. (Even better, enable Travis-CI on your fork and ensure the whole test matrix passes). - [x] Replace "" in the title with the actual Jira issue number, if there is one. - [x] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.txt). --- This is a TransformEvaluator-based implementation of the Window#into primitve, as opposed to the DoFn ProcessContext internals-based implementation. Part of the new Runner API, in which Window.Bound is a primitive. This implementation does not require access to internals within DoFn. You can merge this pull request into a Git repository by running: $ git pull https://github.com/tgroh/incubator-beam ippr_window_primitive Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-beam/pull/149.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #149 ---- commit 3f5762579389d150a89f9e7f871f024daa504430 Author: Thomas Groh Date: 2016-04-08T16:29:31Z Add WindowIntoEvaluatorFactory This is a TransformEvaluator-based implementation of the Window#into primitve, as opposed to the DoFn ProcessContext internals-based implementation. ---- > DirectPipelineRunner: support for unbounded collections > ------------------------------------------------------- > > Key: BEAM-22 > URL: https://issues.apache.org/jira/browse/BEAM-22 > Project: Beam > Issue Type: Improvement > Components: runner-direct > Reporter: Davor Bonaci > Assignee: Thomas Groh > > DirectPipelineRunner currently runs over bounded PCollections only, and implements only a portion of the Beam Model. > We should improve it to faithfully implement the full Beam Model, such as add ability to run over unbounded PCollections, and better resemble execution model in a distributed system. > This further enables features such as a testing source which may simulate late data and test triggers in the pipeline. Finally, we may want to expose an option to select between "debug" (single threaded), "chaos monkey" (test as many model requirements as possible), and "performance" (multi-threaded). > more testing (chaos monkey) > Once this is done, we should update this StackOverflow question: > http://stackoverflow.com/questions/35350113/testing-triggers-with-processing-time/35401426#35401426 -- This message was sent by Atlassian JIRA (v6.3.4#6332)