flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh <jof...@gmail.com>
Subject Re: Flink Application on YARN failed on losing Job Manager | No recovery | Need help debug the cause from logs
Date Fri, 04 Nov 2016 16:09:20 GMT
Thanks, I didn't know about the -z flag!

I haven't been able to get it to work though (using yarn-cluster, with a
zookeeper root configured to /flink in my flink-conf.yaml)

I can see my job directory in ZK under
/flink/application_1477475694024_0015 and I've tried a few ways to restore
the job:

./bin/flink run -m yarn-cluster -yz /application_1477475694024_0015 ....
./bin/flink run -m yarn-cluster -yz application_1477475694024_0015 ....
./bin/flink run -m yarn-cluster -yz /flink/application_1477475694024_0015/
....
./bin/flink run -m yarn-cluster -yz /flink/application_1477475694024_0015
....

The job starts from scratch each time, without restored state.

Am I doing something wrong? I've also tried with -z instead of -yz but I'm
using yarn-cluster to run a single job, so I think it should be -yz.



On Fri, Nov 4, 2016 at 2:33 PM, Ufuk Celebi <uce@apache.org> wrote:

> If the configured ZooKeeper paths are still the same, the job should
> be recovered automatically. On each submission a unique ZK namespace
> is used based on the app ID.
>
> So you have in ZK:
> /flink/app_id/...
>
> You would have to set that manually to resume an old application. You
> can do this via -z flag
> (https://ci.apache.org/projects/flink/flink-docs-
> release-1.2/setup/cli.html).
>
> Does this work?
>
> On Fri, Nov 4, 2016 at 3:28 PM, Josh <jofo90@gmail.com> wrote:
> > Hi Ufuk,
> >
> > I see, but in my case the failure caused YARN application moved into a
> > finished/failed state - so the application itself is no longer running.
> How
> > can I restart the application (or start a new YARN application) and
> ensure
> > that it uses the checkpoint pointer stored in Zookeeper?
> >
> > Thanks,
> > Josh
> >
> > On Fri, Nov 4, 2016 at 1:52 PM, Ufuk Celebi <uce@apache.org> wrote:
> >>
> >> No you don't need to manually trigger a savepoint. With HA checkpoints
> >> are persisted externally and store a pointer in ZooKeeper to recover
> >> them after a JobManager failure.
> >>
> >> On Fri, Nov 4, 2016 at 2:27 PM, Josh <jofo90@gmail.com> wrote:
> >> > I have a follow up question to this - if I'm running a job in
> >> > 'yarn-cluster'
> >> > mode with HA and then at some point the YARN application fails due to
> >> > some
> >> > hardware failure (i.e. the YARN application moves to
> "FINISHED"/"FAILED"
> >> > state), how can I restore the job from the most recent checkpoint?
> >> >
> >> > I can use `flink run -m yarn-cluster -s s3://my-savepoints/id .....`
> to
> >> > restore from a savepoint, but what if I haven't manually taken a
> >> > savepoint
> >> > recently?
> >> >
> >> > Thanks,
> >> > Josh
> >> >
> >> > On Fri, Nov 4, 2016 at 10:06 AM, Maximilian Michels <mxm@apache.org>
> >> > wrote:
> >> >>
> >> >> Hi Anchit,
> >> >>
> >> >> The documentation mentions that you need Zookeeper in addition to
> >> >> setting the application attempts. Zookeeper is needed to retrieve the
> >> >> current leader for the client and to filter out old leaders in case
> >> >> multiple exist (old processes could even stay alive in Yarn).
> Moreover,
> >> >> it
> >> >> is needed to persist the state of the application.
> >> >>
> >> >>
> >> >> -Max
> >> >>
> >> >>
> >> >> On Thu, Nov 3, 2016 at 7:43 PM, Anchit Jatana
> >> >> <development.anchit@gmail.com> wrote:
> >> >> > Hi Maximilian,
> >> >> >
> >> >> > Thanks for you response. Since, I'm running the application on
YARN
> >> >> > cluster
> >> >> > using 'yarn-cluster' mode i.e. using 'flink run -m yarn-cluster
..'
> >> >> > command.
> >> >> > Is there anything more that I need to configure apart from setting
> up
> >> >> > 'yarn.application-attempts: 10' property inside
> conf/flink-conf.yaml.
> >> >> >
> >> >> > Just wished to confirm if there is anything more that I need to
> >> >> > configure to
> >> >> > set up HA on 'yarn-cluster' mode.
> >> >> >
> >> >> > Thank you
> >> >> >
> >> >> > Regards,
> >> >> > Anchit
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > View this message in context:
> >> >> >
> >> >> > http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/Flink-Application-on-YARN-failed-on-losing-Job-Manager-No-
> recovery-Need-help-debug-the-cause-from-los-tp9839p9887.html
> >> >> > Sent from the Apache Flink User Mailing List archive. mailing
list
> >> >> > archive at Nabble.com.
> >> >
> >> >
> >
> >
>

Mime
View raw message