Return-Path: X-Original-To: apmail-apex-dev-archive@minotaur.apache.org Delivered-To: apmail-apex-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 8C7D418C0D for ; Wed, 23 Sep 2015 15:39:56 +0000 (UTC) Received: (qmail 61308 invoked by uid 500); 23 Sep 2015 15:39:56 -0000 Delivered-To: apmail-apex-dev-archive@apex.apache.org Received: (qmail 61252 invoked by uid 500); 23 Sep 2015 15:39:56 -0000 Mailing-List: contact dev-help@apex.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@apex.incubator.apache.org Delivered-To: mailing list dev@apex.incubator.apache.org Received: (qmail 61226 invoked by uid 99); 23 Sep 2015 15:39:56 -0000 Received: from Unknown (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Sep 2015 15:39:56 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id A82DFC0252 for ; Wed, 23 Sep 2015 15:39:55 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.673 X-Spam-Level: X-Spam-Status: No, score=0.673 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.006, SPF_HELO_PASS=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=capitalone.com header.b=WUDWZm3M; dkim=pass (2048-bit key) header.d=capitalone.com header.b=T9O1ghUX Received: from mx1-us-east.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id rpO0J8nvmBWA for ; Wed, 23 Sep 2015 15:39:49 +0000 (UTC) Received: from outp.capitalone.com (outp.capitalone.com [204.63.55.164]) by mx1-us-east.apache.org (ASF Mail Server at mx1-us-east.apache.org) with ESMTPS id 1649C44189 for ; Wed, 23 Sep 2015 15:39:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=capitalone.com; l=2195; q=dns/txt; s=SM2048Mar2015P; t=1443022789; x=1443109189; h=from:to:date:subject:message-id:references:in-reply-to: mime-version:content-transfer-encoding; bh=0sO/LB/PrORbJI0VsKN1lnpzor9YdWlQHPUjsfnLZjU=; b=WUDWZm3M/pEMRvjU2Se8Qj3Kqv0627nQ4sdw9HI8o6oG06+CdUyY46g+ jqUSAeM/dabD3aDzfeR+HKDLwPXv6S72f8tQHAnaGNKOJrirSHFqPTALo CFE4F2jBobuhh18SEdVe2hum5/TuIVJwR5wc5H9PwHqJmD8Wn0iE2nmbh cTNCVNPjGrZcFsJKHZXvO9ewJ7D9SAYnz2S6l8Z5DpC9xDCfVX7jGUabS vNgCkmprOk+faxO9SXmTnHwDy4iRAvkOoFXRnkx2cBIpfcGg8SdKQgVNN kmSnwmlWVwMj4RczmuzRQxQAtuXirvofc5RZztMm9zlBlU3DAkZeiCpTR g==; X-IronPort-AV: E=McAfee;i="5700,7163,7932"; a="94200542" X-IronPort-AV: E=Sophos;i="5.17,577,1437451200"; d="scan'208";a="94200542" X-AFNotificationDomain: TRUE Received: from komail01.kdc.capitalone.com ([10.37.137.132]) by pomail02.capitalone.com with ESMTP/TLS/ADH-CAMELLIA256-SHA; 23 Sep 2015 11:39:48 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=capitalone.com; l=1422; q=dns/txt; s=SM2048Mar2015K; t=1443022788; x=1443109188; h=from:to:date:subject:message-id:references:in-reply-to: mime-version:content-transfer-encoding; bh=1W82u8RLTmouqWgSDBM8mx4OUS/H/7oQTvnFf53Ty+Y=; b=T9O1ghUXDKxclgmlolg5e6aCXf2tjM07Xa8ORODpB+oJgjQMkckafDsR Zxi1djLCIE66YJzCeHA22QKwmq/GxOCrxVxT1aJvIgSlMLC/nyS7XOkds mmM7rastz+PRxYbeiAIY1cvfR8SWor7GxfvWkO1B53fCScrIW31MnfViu t3sqHdGhbtSJYdB21axy/aELuMniMWB5ru8IMi6fbRbAPspRfssMVF0+G /xorpWeZ3T1AuCLmh/DJ/rxJ13ripTVK5nnZMyF/5bc2RAxRuAIWUtKv8 MaDDfuOwsC8/kL/FEC9iWMVq7sXPGLu2kDYhVNXGdC3Cm5lYvCJs+R/Ky Q==; X-IronPort-AV: E=McAfee;i="5700,7163,7932"; a="278801151" X-IronPort-AV: E=Sophos;i="5.17,577,1437451200"; d="scan'208";a="278801151" X-AFNotificationDomain: TRUE Received: from kdcpexcasht02.cof.ds.capitalone.com ([10.37.194.12]) by komail01.kdc.capitalone.com with ESMTP/TLS/AES128-SHA; 23 Sep 2015 11:39:48 -0400 Received: from KDCPEXCMB01.cof.ds.capitalone.com ([169.254.1.126]) by KDCPEXCASHT02.cof.ds.capitalone.com ([fe80::f13f:d6f2:e567:c056%16]) with mapi; Wed, 23 Sep 2015 11:39:47 -0400 From: "Ganelin, Ilya" To: "dev@apex.incubator.apache.org" Date: Wed, 23 Sep 2015 11:39:51 -0400 Subject: Re: Support easy parallelization of pipelines Thread-Topic: Support easy parallelization of pipelines Thread-Index: AdD2FhNlopg7IRH/QG+/wDs3G0IFSg== Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.5.3.150624 acceptlanguage: en-US Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Ram, thank you. I think this is a good starting point, however it requires having access to the stream at creation time (as well as the operator being added). I=B9d ideally like to create a function: static void parallelize(DAG dag); This function would take the head of the DAG, and parallelize all downstream operators. It looks like at the moment, there is no interface within DAG to provide access to its operators or streams. Does such an interface exist or is this something I would need to expose? Was there a design decision to not expose these? On 9/23/15, 11:26 AM, "Munagala Ramanath" wrote: >l ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary= to Capital One and/or its affiliates and may only be used solely in perfor= mance of work or services for Capital One. The information transmitted here= with is intended only for use by the individual or entity to which it is ad= dressed. If the reader of this message is not the intended recipient, you a= re hereby notified that any review, retransmission, dissemination, distribu= tion, copying or other use of, or taking of any action in reliance upon thi= s information is strictly prohibited. If you have received this communicati= on in error, please contact the sender and delete the material from your co= mputer. ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary= to Capital One and/or its affiliates and may only be used solely in perfor= mance of work or services for Capital One. The information transmitted here= with is intended only for use by the individual or entity to which it is ad= dressed. If the reader of this message is not the intended recipient, you a= re hereby notified that any review, retransmission, dissemination, distribu= tion, copying or other use of, or taking of any action in reliance upon thi= s information is strictly prohibited. If you have received this communicati= on in error, please contact the sender and delete the material from your co= mputer.