apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Isha Arkatkar <i...@datatorrent.com>
Subject Re: Check pointing question
Date Fri, 19 Feb 2016 01:39:59 GMT
Hi Jim,

    In the case when input operator was killed and then restarted, all the
downstream operators are also restarted from previous available checkpoint.
    So answer to first question is: No, the downstream operators do not
continue processing.
    And the tuples which were already in flow at the router level are
actually not processed. As, those tuples would be sent again since the data
is processed again from previous checkpoint.

Thanks!
Isha

On Thu, Feb 18, 2016 at 2:20 PM, Jim <jim@facility.supplies> wrote:

> I have an interesting question, on a new Apex application that I am
> writing (note this is my first Apex application, so my wording of it may
> not be in all the correct terminology – please correct me if I have not
> stated something in the right way).
>
>
>
> I am getting in data transmissions, via a streaming data pipe, that can
> each contain 1 or 500 individual transactions contained within them.
>
>
>
> My current flow has this data coming into an input operator, which looks
> at the individual transactions contained with each transmission, and
> depending on transaction type, emits a tuple containing an individual
> transaction, and hands it off to an appropriate transaction specific
> operator for processing.
>
>
>
> If the main input operator, while doing this splitting and emitting of
> transactions to the various transaction specific operators encounters a
> system problem where it suddenly stops processing, I have some questions:
>
>
>
> 1.)    Will each of the downstream operators continue to process what has
> already been emitted to them and continue to move them through the system?
>
> 2.)    If the answer to #1 above is yes, they will continue to process,
> how do I checkpoint what transaction point I am at within my application,
> so that when it restarts it will skip the xx records that it already handed
> off before the application went down, and resume processing at the record
> that was not yet emitted to any of the transaction specific operators for
> processing?
>
> 3.)    If the answer is no to #1 above, what does it do to all the tuples
> that are already in flow below the router level?
>
>
>
> Thanks for any help and information that you can provide.
>
>
>
> Thanks,
>
>
>
> Jim
>

Mime
View raw message