flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sridhar Chellappa <flinken...@gmail.com>
Subject Re: data loss after implementing checkpoint
Date Wed, 19 Jul 2017 04:38:17 GMT
A follow up question on this. I have a Complex Event processor implemented
using the CEP library (1.3.0). The CEP library runs a variety of rules that
are configured (enable/disable rule) VIA REST APIs.

Now, if my application crashes and recovers (or is cancelled and
restarted), will my configuration(as to which rules are enabled) still
hold? or do I have to persist the info into a backend?


On Mon, Jul 10, 2017 at 7:36 PM, Nico Kruber <nico@data-artisans.com> wrote:

> Hi Aftab,
> looks like what you want is either an externalized checkpoint with
> RETAIN_ON_CANCELLATION mode [1] or a savepoint [2].
>
> Ordinary checkpoints are deleted when the job is cancelled and only serve
> as a
> fault tolerance layer in case something goes wrong, i.e. machines fail, so
> that the job can be restarted automatically based on the restart policy.
>
>
> Nico
>
> [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/
> checkpoints.html
> [2] https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/
> savepoints.html
>
> On Monday, 10 July 2017 14:20:53 CEST Aftab Ansari wrote:
> > Hi,
> > I am new to flink. I am facing issue implementing checkpoint.
> >
> > checkpoint related code:
> >
> > long checkpointInterval = 5000;
> >
> >  StreamExecutionEnvironment env = StreamUtils.getEnvironment(params);
> >  //specify backend
> >  //env.setStateBackend(new FsStateBackend("s3n://xxx/flink-state/"),
> true));
> > env.setStateBackend(new
> > FsStateBackend("file:///Users/aftabansari/flink-state/", true));
> >  //enable checkpoint
> >  env.enableCheckpointing(checkpointInterval,
> > CheckpointingMode.EXACTLY_ONCE);
> > env.getCheckpointConfig().setMinPauseBetweenCheckpoints(500);
> >
> >
> > When I run the code, I can see flink-state being written in my local
> > machine. but when I stop the job , wait for a few minutes and restart the
> > job, it does not pick up from the time it left but it starts from when I
> > started the job.
> >
> > Could you point out what i am doing wrong. I am testing it locally from
> > ideaIntellij. below is what i see from localhost. Any help would be
> > appreciated. Thanks
> > [image: Inline images 1]
> > Br,
>
>

Mime
View raw message