apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ashwin Chandra Putta <ashwinchand...@gmail.com>
Subject Re: A few questions about operator
Date Tue, 19 Apr 2016 22:08:30 GMT
Hey Ananth,

Please find the answers inline.


On Tue, Apr 19, 2016 at 12:20 PM, Ananth Gundabattula <
agundabattula@gmail.com> wrote:

> Hello All,
> I was wondering if you could help me with the following questions as I was
> not able to locate the info from the docs:
>    - When is the constructor of an operator called ? The docs say once in
>    the lifetime of an operator but I was wondering whether the definition of
>    "lifetime" spans across start/stop/crash ( because of a coding error ) of
>    an apex application ?
> [Ashwin] The operator has a life cycle. Constructor  --> checkpointed
state applied --> setup() --> loop {beginWindow() --> loop {process() for
each input port} --> endWindow()} --> tearDown(). When an operator
recovers, the cycle is the same. The checkpointed state is the state from
last known checkpoint before the operator crashed.

>    - Regarding backpressure and the buffer server, how does the buffer
>    server survive application crashes ? I mean considering a situation when
>    the bufferserver itself dies for whatever reason, is it guaranteed a
>    downstream operator will eventually catchup with an Upstream operator when
>    the buffer server is brought back up?
> [Ashwin] The buffer server always lives with the upstream operator. In
fact, it lives within the same JVM as upstream operator. If an operator
fails, the upstream operator's buffer server will have the required data
state. If the upstream operator fails, its upstream operator's buffer
server has the data state and so on.

>    - Is there an equivalent of the OffsetManager for the 0.9 version of
>    the Kafka operator ?
>    - Am I correct in assuming that the moment we rename an application,
>    the semantics of the Kafka operator will completely change and might end up
>    reading from the "initialOffset" by the application code ? How is the
>    semantics maintained for a definition of "application name" ? Does every
>    deploy of the application code result in a new application or it is simply
>    using the @ApplicationAnnotation(name="") instance to define this meaning ?
> [Ashwin] You can launch an application from its previous state by using
the -originalAppId parameter and provide the yarn application id from its
previous run's checkpointed state, it should apply to all operator within
the dag including the kafka input operator. You can also provide a new name
for the application using attribute dt.attr.APPLICATION_NAME. eg: launch
pi-demo-3.4.0-incubating-SNAPSHOT.apa -originalAppId
application_1459879799578_8727 -Ddt.attr.APPLICATION_NAME="pidemo v201"

> Thanks for your time.
> Regards,
> Ananth



View raw message