apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Farkas <...@datatorrent.com>
Subject Re: Support easy parallelization of pipelines
Date Wed, 23 Sep 2015 15:48:00 GMT
Hi Ilya,

There are two ways to do this. You can do normal N x N partitioning, but
write a partitioner for B which assigns the same partition keys to M
operators in a partitioning. Then M partitions will receive the same data:
An example of setting partition keys in a partitioner is here:

https://github.com/apache/incubator-apex-core/blob/devel-3/common/src/main/java/com/datatorrent/common/partitioner/StatelessPartitioner.java

An easier way is this:

dag.addStream(“NAME”, A.output, B.input, C.input);

You have to put all the input ports you want to receive the same data in
the same stream declaration.

Tim

On Wed, Sep 23, 2015 at 7:39 AM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com>
wrote:

> Ram, thank you. I think this is a good starting point, however it requires
> having access to the stream at creation time (as well as the operator
> being added). I¹d ideally like to create a function:
>
> static void parallelize(DAG dag);
>
> This function would take the head of the DAG, and parallelize all
> downstream operators. It looks like at the moment, there is no interface
> within DAG to provide access to its operators or streams. Does such an
> interface exist or is this something I would need to expose? Was there a
> design decision to not expose these?
>
>
> On 9/23/15, 11:26 AM, "Munagala Ramanath" <ram@datatorrent.com> wrote:
>
> >l
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message