flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Till Rohrmann <trohrm...@apache.org>
Subject Re: YARN session application attempts
Date Tue, 19 Apr 2016 17:36:00 GMT
Hi Stefano,

Hadoop supports this feature since version 2.6.0. You can define a time
interval for the maximum number of applications attempt. This means that
you have to observe this number of application failures in a time interval
before failing the application ultimately. Flink will activate this feature
if you’re using Hadoop >= 2.6.0. The failures validity interval will be set
to the akka.ask.timeout value (default: 10s).

[1]
https://hadoop.apache.org/docs/r2.6.0/api/org/apache/hadoop/yarn/api/records/ApplicationSubmissionContext.html#setAttemptFailuresValidityInterval(long
)

Cheers,
Till
​

On Tue, Apr 12, 2016 at 11:56 AM, Stefano Baghino <
stefano.baghino@radicalbit.io> wrote:

> Hi Ufuk, sorry for taking an awful lot of time to reply but I fell behind
> with the ML in the last couple of weeks due to lack of time.
> First of all, thanks for taking the time to help me.
>
> Yes, what I was saying was that apparently from the code (and effectively
> as we later found out after a couple of tests) the "upper bound" cited by
> the documentation seems invalid (meaning, the number of attempts if
> effectively regulated by the flink-conf.yaml and falls back to the
> yarn-site.xml only if missing).
>
> We're currently using Hadoop 2.7.1 so we'll try your solution, thanks.
>
> I was also wondering if there's a way to ask to retry indefinitely, so
> that a long-running streaming job can endure as many job manager failures
> as possible without ever needing for a human intervention to restart the
> YARN session.
>
> On Sat, Apr 2, 2016 at 3:53 PM, Ufuk Celebi <uce@apache.org> wrote:
>
>> Hey Stefano,
>>
>> yarn.resourcemanager.am.max-attempts is a setting for your YARN
>> cluster and cannot be influenced by Flink. Flink cannot set a higher
>> number than this for yarn.application-attempts.
>>
>> The key that is set/overriden by Flink is probably only valid for the
>> YARN session, but I'm not too familiar with the code. Maybe someone
>> else can chime in.
>>
>> I would recommend using a newer Hadoop version (>= 2.6), where you can
>> configure the failure validity interval, which counts the attempts per
>> time interval, e.g. it is allowed to fail 2 times within X seconds.
>> Per default, the failure validity interval is configured to the Akka
>> timeout (which is per default 10s). I actually think it would make
>> sense to increase this a little and leave the attempts at 1 or 2 (in
>> the interval).
>>
>> Does this help?
>>
>> – Ufuk
>>
>>
>> On Fri, Apr 1, 2016 at 3:24 PM, Stefano Baghino
>> <stefano.baghino@radicalbit.io> wrote:
>> > Hello everybody,
>> >
>> > I was asking myself: are there any best practices regarding how to set
>> the
>> > `yarn.application-attempts` configuration key when running Flink on
>> YARN as
>> > a long-running session? The configuration page on the docs states that
>> 1 is
>> > the default and that it is recommended to leave it like that, however
>> in the
>> > case of a long running session it seems to me that the value should be
>> > higher in order to actually allow the session to keep running despite
>> Job
>> > Managers failing.
>> >
>> > Furthermore, the HA page on the docs states the following
>> >
>> > """
>> > It’s important to note that yarn.resourcemanager.am.max-attempts is an
>> upper
>> > bound for the application restarts. Therfore, the number of application
>> > attempts set within Flink cannot exceed the YARN cluster setting with
>> which
>> > YARN was started.
>> > """
>> >
>> > However, after some tests conducted by my colleagues and after looking
>> at
>> > the code (FlinkYarnClientBase:522-536) it seems to me that the
>> > flink-conf.yaml key, if set, overrides the yarn-site.xml, which in turn
>> > overrides the fallback value of 1. Is this right? Is the documentation
>> > wrong?
>> >
>> > --
>> > BR,
>> > Stefano Baghino
>> >
>> > Software Engineer @ Radicalbit
>>
>
>
>
> --
> BR,
> Stefano Baghino
>
> Software Engineer @ Radicalbit
>

Mime
View raw message