apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <...@apache.org>
Subject Re: Partitioning information and current operators Id in setup()/activate() method.
Date Thu, 21 Sep 2017 15:33:53 GMT
Implementing Partitioner should cover this. The partitioner is called when
constructing the physical plan (at initialization) and then, in the case of
dynamic partitioning, on demand.

As part of the partitioning logic you can set properties on the operator
instances that are specific to the partition. An example for this is the
Kafka input operator, where each Apex partition will be setup with the
Kafka topic partition(s) it is responsible for.

Thanks,
Thomas


On Thu, Sep 21, 2017 at 6:36 AM, Ananth G <ananthg.apex@gmail.com> wrote:

> Hello All,
>
> I was wondering if there is a good way to get the following two values in
> an operator when the operator is in the activate/setup methods
>
> 1. The total number of operator physical instances that are going to be
> launched
> 2. The ordinal position of the current operator in the current set of
> physical operators
>
> I am looking for this information so that I can implement a logic in the
> operator wherein each operator takes a part of the responsibility . I can
> get the above values when the dynamic partitioning interface is implemented
> and the call back happens to the assign() method. But the issue is that the
> assign() method does not seem to be invoked at the beginning but only when
> the dynamic partitioning is being done ( In other words the assign() method
> is not called until a certain time has passed since the beginning of the
> launch of the operator. I would like the above values when the operator is
> starting in the activate()/setup() methods.)
>
> Any advice as to how these values can be obtained ?
>
> Regards,
> Ananth

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message