apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Munagala Ramanath <...@datatorrent.com>
Subject Re: Sequencing of operator calls
Date Tue, 10 Nov 2015 18:33:30 GMT
Chetan's answer provides a good explanation as well as clarifying that
the difference can be more than 1.

Since checkpointing (i.e. "commit notification" as Thomas refers to
it) is asynchronous, I'm curious
about whether the window ids in the checkpointed call are guaranteed
to be sequential or if they could
be out of order, i.e. can the checkpointed call see window id 101
before it sees 100 ?

Ram

On Tue, Nov 10, 2015 at 10:27 AM, Bhupesh Chawda
<bhupesh@datatorrent.com> wrote:
> Hi Tim,
> Thanks for the detailed explanation.
> I understand that the sequence would be
> beginWindow  (x) -> endWindow (x) -> checkpointed (x)  -> beginWindow
> (x+1)
>
> However when I try to print out the window ids in beginWindow, endWindow
> and checkpointed calls,  I see x and x-1 respectively.
> I.e. If the window just before checkpoint is 100, I see that the
> checkpointed call had window id 99.
>
> Note: This is observed in the local mode of Apex.
>
> Thanks
> -Bhupesh
> On 10-Nov-2015 11:25 pm, "Timothy Farkas" <tim@datatorrent.com> wrote:
>
>> Hi Bhupesh,
>>
>> The sequencing of checkpoint called in relation to beginWindow and
>> endWindow depends on how your APPLICATION_WINDOW_COUNT and
>> CHECKPOINT_WINDOW_COUNT are configured. If the two WINDOW_COUNTs are not
>> configured to be the same then it is possible that checkpointed is called
>> within an application window. So the sequence of events would be this:
>>
>> beginWindow -> checkpointed -> endWindow
>>
>> If the APPLICATION_WINDOW_COUNT and CHECKPOINT_WINDOW_COUNT are the same
>> (default). Then the sequence of calls would be this:
>>
>> beginWindow  (100) -> endWindow (100) -> checkpointed (100)  -> beginWindow
>> (101)
>>
>> The numbers in the sequence represent possible streaming window Ids that
>> each call would be associated with.
>>
>> The StateMachine which calls these callbacks for non-input operators is in
>> GenericNode.java.
>>
>> Tim
>>
>> On Tue, Nov 10, 2015 at 3:36 AM, Bhupesh Chawda <bhupesh@datatorrent.com>
>> wrote:
>>
>> > Hi Chetan / Community,
>> >
>> > Can someone please elaborate on why the window id supplied to
>> > CheckpointListener and the Operator would differ.
>> > I tried looking at the window ids of checkpointed() and the beginWindow()
>> > calls and they differ by 1. Don't know why this should be the case.
>> >
>> > Thanks.
>> > -Bhupesh
>> >
>> > On Thu, Sep 17, 2015 at 5:56 AM, Chetan Narsude <chetan@datatorrent.com>
>> > wrote:
>> >
>> > > Short answer is yes.
>> > >
>> > > All the control tuples are scheduled to be delivered outside of the
>> > window.
>> > > As checkpointed callback is triggered because of CHECKPOINT control
>> > tuple,
>> > > it will happen after endWindow and before the next beginWindow.
>> > >
>> > > The windowId supplied to CheckpointListener and the one provided to
>> > > Operator need not match even though the sequence is defined. So I am
>> > > curious how you intend to use this knowledge.
>> > >
>> > > --
>> > > Chetan
>> > >
>> > >
>> > > On Tue, Sep 15, 2015 at 8:31 AM, Thomas Weise <thomas@datatorrent.com>
>> > > wrote:
>> > >
>> > > > It has not changed the operator execution model. State serialization
>> is
>> > > > still synchronous, write to HDFS is taken out of the operator thread.
>> > > >
>> > > > On Tue, Sep 15, 2015 at 8:18 AM, Amol Kekre <amol@datatorrent.com>
>> > > wrote:
>> > > >
>> > > > >
>> > > > > Sent too soon. Has asynchronous checkpointing changed this?
>> > > > >
>> > > > > Amol
>> > > > >
>> > > > > Sent from my iPhone
>> > > > >
>> > > > > > On Sep 15, 2015, at 12:38 AM, Bhupesh Chawda <
>> > > bhupesh@datatorrent.com>
>> > > > > wrote:
>> > > > > >
>> > > > > > Hi All,
>> > > > > >
>> > > > > > Is it safe to assume that the checkpointed() and the
>> beginWindow()
>> > > > calls
>> > > > > > are sequenced?
>> > > > > > In other words, are these calls part of the same thread
and may
>> not
>> > > run
>> > > > > in
>> > > > > > parallel?
>> > > > > >
>> > > > > > Thanks.
>> > > > > >
>> > > > > > --
>> > > > > > -Bhupesh
>> > > > >
>> > > >
>> > >
>> >
>>

Mime
View raw message