apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Farkas <...@datatorrent.com>
Subject Re: Is there a way to provide a non-transient field that can also skip checkpoint?
Date Tue, 05 Jan 2016 21:10:47 GMT
A small extension to the default Kryo field serializer could be made to
ignore fields with a particular annotation, as mentioned in the google
group below:

https://groups.google.com/forum/#!topic/kryo-users/nWSni1JVFKY


On Tue, Jan 5, 2016 at 11:08 AM, Gaurav Gupta <gaurav@datatorrent.com>
wrote:

> Siyuan,,,
>
> Questions inline
>
> Thanks
> - Gaurav
>
> > On Jan 5, 2016, at 11:02 AM, Siyuan Hua <siyuan@datatorrent.com> wrote:
> >
> > I can think of 3 use case:
> > 1. I want to expose some runtime property through REST API (isAlive,
> > isWaiting etc)
>
> When you say runtime properties, do they not automatically qualify for
> being transient?
>
> > 2. Some properties need to be set only once at the beginning of the
> > application based on some condition (we don't need checkpoint such kind
> of
> > properties)
>
> How would recovery work?
>
> > 3. Some properties are calculated in AM(definePartition etc.). They are
> not
> > necessarily checkpointable but need to be serialized to individual
> instance.
> >
> Again same question, how would recovery work?
>
> > It may only have negligible improvement in performance, but it helps to
> > understand the scope of the state of the operator: Not all non-transient
> > state needs to be persistent.
> >
> >
> > Any thoughts?
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message