apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ganelin, Ilya" <Ilya.Gane...@capitalone.com>
Subject Support easy parallelization of pipelines
Date Wed, 23 Sep 2015 15:06:17 GMT
Hi all. Say that I have a DAG (A -> B -> C) and, given any DAG, I want to create a new
DAG that is (A -> B -> C,  A -> B’ -> C’)

At the moment, Apex does not support the following operation:

dag.addStream(“NAME”, A.output, B.input)
dag.addStream(“NAME2”, A.output, B’.input)

It throws the following error:
java.lang.IllegalArgumentException: Operator A already connected to NAME

One has to do :
dag.addStream(“name”, A.output,  B.input, B’.input).

Is there a concrete reason that the first approach is not supported? Is there any way to enable
such an approach to allow iterative building of graphs? This would allow me to create a utility
that automatically generates independent parallel pipelines given the same point of origin
which greatly facilitates low-latency fault handling.

Is there a reason an operator may not be connected to multiple operators after initialization?

Appreciate any input, thanks!
________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message