apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <tho...@datatorrent.com>
Subject Re: Why is Async checkpointing made default?
Date Fri, 27 Nov 2015 23:00:43 GMT
Chetan,

Would like to understand how the checkpointed callback helps you with what
you indicated. This may require a specific example. Let's take it offline.

Thomas

On Wed, Nov 25, 2015 at 4:07 PM, Chetan Narsude (cnarsude) <
cnarsude@cisco.com> wrote:

> Yes - a few but cannot share the details - protected under NDA - ping me
> in private and I can probably be able to give you more generic details on
> similar cooked up examples.
>
>  The part that follows “e.g.” below is an example that probably is
> sufficient to infer the use case logically, I think. I shared that to
> exemplify how changing the semantics will break semver.
>
> —
> Chetan
>
>
>
>
>
>
>
>
>
>
>
> On 11/25/15, 3:51 PM, "Thomas Weise" <thomas@datatorrent.com> wrote:
>
> >Do you have a specific example?
> >
> >I see this happening in committed(), but not in checkpointed() where the
> >checkpoint remains intermediate, whether it was copied to HDFS or not.
> >
> >
> >On Wed, Nov 25, 2015 at 3:42 PM, Chetan Narsude (cnarsude) <
> >cnarsude@cisco.com> wrote:
> >
> >> >
> >> >Until we have this, how about we restore the previous behavior
> >> >temporarily?
> >> >Calling checkpointed() immediately does not seem to pose any practical
> >> >issue but ensures that the code that was written under this assumption
> >>is
> >> >not broken.
> >>
> >> We can¹t do it. It would be incorrect. It breaks all the other code that
> >> (unassumingly) correctly complied to the semantics. e.g. an operator
> >>which
> >> informs interesting parties that the checkpointed data is available for
> >> immediate consumption from storage.
> >>
> >> ‹
> >> Chetan
> >>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message