apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <tho...@datatorrent.com>
Subject Re: Sequencing of operator calls
Date Sat, 19 Sep 2015 17:39:54 GMT
There seems to be confusion between "checkpointed" and "committed". I put
my comment on that JIRA.

Serializing the state of the operator is always blocking. Write to HDFS can
be asynchronous. Commit notification is always asynchronous.

Thomas

On Sat, Sep 19, 2015 at 10:19 AM, Amol Kekre <amol@datatorrent.com> wrote:

> Bhupesh,
> If you want to guaranteed that checkpoint is fully done before next tuple
> is processed, you will need to use blocking checkpoint attribute.
>
> Thks
> Amol
>
> On Sat, Sep 19, 2015 at 8:59 AM, Sandesh Hegde <sandesh@datatorrent.com>
> wrote:
>
> > Blocking checkpoint is supported, please refer to the following jira,
> > https://malhar.atlassian.net/browse/APEX-19
> >
> > On Fri, Sep 18, 2015 at 9:08 PM Bhupesh Chawda <bhupesh@datatorrent.com>
> > wrote:
> >
> > > Thanks for the replies.
> > > My intention was to ask if the call to next begin window will start
> only
> > > after the checkpointed call has done it's job.
> > > So if just the call is sequenced, but it's a different thread, then it
> > does
> > > not really help in my case. I need to finish the job which I need to do
> > in
> > > checkpointed call before any new tuples are processed.
> > > On 17-Sep-2015 5:56 am, "Chetan Narsude" <chetan@datatorrent.com>
> wrote:
> > >
> > > > Short answer is yes.
> > > >
> > > > All the control tuples are scheduled to be delivered outside of the
> > > window.
> > > > As checkpointed callback is triggered because of CHECKPOINT control
> > > tuple,
> > > > it will happen after endWindow and before the next beginWindow.
> > > >
> > > > The windowId supplied to CheckpointListener and the one provided to
> > > > Operator need not match even though the sequence is defined. So I am
> > > > curious how you intend to use this knowledge.
> > > >
> > > > --
> > > > Chetan
> > > >
> > > >
> > > > On Tue, Sep 15, 2015 at 8:31 AM, Thomas Weise <
> thomas@datatorrent.com>
> > > > wrote:
> > > >
> > > > > It has not changed the operator execution model. State
> serialization
> > is
> > > > > still synchronous, write to HDFS is taken out of the operator
> thread.
> > > > >
> > > > > On Tue, Sep 15, 2015 at 8:18 AM, Amol Kekre <amol@datatorrent.com>
> > > > wrote:
> > > > >
> > > > > >
> > > > > > Sent too soon. Has asynchronous checkpointing changed this?
> > > > > >
> > > > > > Amol
> > > > > >
> > > > > > Sent from my iPhone
> > > > > >
> > > > > > > On Sep 15, 2015, at 12:38 AM, Bhupesh Chawda <
> > > > bhupesh@datatorrent.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi All,
> > > > > > >
> > > > > > > Is it safe to assume that the checkpointed() and the
> > beginWindow()
> > > > > calls
> > > > > > > are sequenced?
> > > > > > > In other words, are these calls part of the same thread
and may
> > not
> > > > run
> > > > > > in
> > > > > > > parallel?
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > --
> > > > > > > -Bhupesh
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message