apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munagala Ramanath <...@datatorrent.com>
Subject Re: Support easy parallelization of pipelines
Date Wed, 23 Sep 2015 15:26:53 GMT
The returned StreamMeta object has an addSink() method for additng
additional sinks.


On Wed, Sep 23, 2015 at 8:06 AM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com>

> Hi all. Say that I have a DAG (A -> B -> C) and, given any DAG, I want to
> create a new DAG that is (A -> B -> C,  A -> B’ -> C’)
> At the moment, Apex does not support the following operation:
> dag.addStream(“NAME”, A.output, B.input)
> dag.addStream(“NAME2”, A.output, B’.input)
> It throws the following error:
> java.lang.IllegalArgumentException: Operator A already connected to NAME
> One has to do :
> dag.addStream(“name”, A.output,  B.input, B’.input).
> Is there a concrete reason that the first approach is not supported? Is
> there any way to enable such an approach to allow iterative building of
> graphs? This would allow me to create a utility that automatically
> generates independent parallel pipelines given the same point of origin
> which greatly facilitates low-latency fault handling.
> Is there a reason an operator may not be connected to multiple operators
> after initialization?
> Appreciate any input, thanks!
> ________________________________________________________
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message