apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Narsude <che...@datatorrent.com>
Subject Re: Thread Local and Container Local
Date Thu, 27 Aug 2015 21:20:33 GMT
Hi Brennon,

  To speak in terms of concrete example you gave (thanks for that):

Broadly speaking, the reasons for partitioning can be:

1. You want to use more resources to execute your business logic (divide
and conquer).
2. You want to simplify your logic by segregating the dataset.


Although 2 is a valid reason, it can easily be achieved by providing some
boilerplate code in your business logic. So we will mostly focus on reason
1.

So when X needs to be partitioned, you are explicitly stating that if
possible you need different resources for each of the partitions. The same
is true for Y. So Apex will try to allocate separate containers for each of
those instances.

So the idea of Y instance being container/thread local to X has no
possibility at the logical level.

However when X gets partitioned 2 ways, each of the Y partition now needs
unifier which unifies 2 streams, one coming from each of X instances. This
unifier though now can be in the same container or the thread as instance
of Y. And that's the resolution Apex resorts to.

TLDR; the locality is applied to the stream between the unifier and the
downstream instance.


com.datatorrent.stram.plan.physical.PhysicalPlan.class has the code -
specifically follow the trails from setLocalityGrouping method.

--
Chetan

On Thu, Aug 27, 2015 at 10:59 AM, York, Brennon <Brennon.York@capitalone.com
> wrote:

> 1. Does anyone have any sample code (or can point me to some) where we
> demonstrate thread local and container local operators? Answered my own
> question, check this out<
> https://github.com/apache/incubator-apex-core/blob/bdd7109519453e67789e4ec4025092a977d2b27c/engine/src/test/java/com/datatorrent/stram/stream/OiOStreamTest.java#L64
> >.
> 2. When doing thread or container local streams how does that work with
> dynamic (or differing sized) partitions between the two operators?
> Concretely, if I have a logical plan that looks like:
>
> X[1] => Y[1]
>
> And the physical plan looks like:
>
> X[2] => Y[16]
>
> How does the grouping work? Would it put 1 physical X operator and 8
> physical Y operators in one grouping and the other set of physical
> operators in another grouping or does it do something else? And, for
> edification, where does the Apex code reside that does this work?
>
> Thanks all!
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message