Return-Path: X-Original-To: apmail-flink-user-archive@minotaur.apache.org Delivered-To: apmail-flink-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id EBFEE1977A for ; Mon, 21 Mar 2016 14:30:35 +0000 (UTC) Received: (qmail 56144 invoked by uid 500); 21 Mar 2016 14:30:35 -0000 Delivered-To: apmail-flink-user-archive@flink.apache.org Received: (qmail 56045 invoked by uid 500); 21 Mar 2016 14:30:35 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 56035 invoked by uid 99); 21 Mar 2016 14:30:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 21 Mar 2016 14:30:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 127941800EC for ; Mon, 21 Mar 2016 14:30:35 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -2.001 X-Spam-Level: X-Spam-Status: No, score=-2.001 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RP_MATCHES_RCVD=-0.001] autolearn=disabled Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id QVLNPWie8BBo for ; Mon, 21 Mar 2016 14:30:33 +0000 (UTC) Received: from mail2-relais-roc.national.inria.fr (mail2-relais-roc.national.inria.fr [192.134.164.83]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id A54145F246 for ; Mon, 21 Mar 2016 14:30:32 +0000 (UTC) X-IronPort-AV: E=Sophos;i="5.24,372,1454972400"; d="scan'208,217";a="209248518" Received: from ovidcmarcu.irisa.fr ([131.254.17.6]) by mail2-relais-roc.national.inria.fr with ESMTP/TLS/DHE-RSA-AES256-SHA; 21 Mar 2016 15:30:24 +0100 From: Ovidiu-Cristian MARCU Content-Type: multipart/alternative; boundary="Apple-Mail=_39783C26-9203-41A4-B755-68A4940B005A" Message-Id: <12C83C3D-4CAC-4ED3-BD37-689CF06C9B28@inria.fr> Mime-Version: 1.0 (Mac OS X Mail 9.2 \(3112\)) Subject: Re: Not enough free slots to run the job Date: Mon, 21 Mar 2016 15:30:24 +0100 References: <8D0BD035-14C6-4E05-919F-9F0692F927F0@inria.fr> To: user@flink.apache.org In-Reply-To: X-Mailer: Apple Mail (2.3112) --Apple-Mail=_39783C26-9203-41A4-B755-68A4940B005A Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Robert, I am not sure I understand so please confirm if I understand correctly = your suggestions: - to use less slots than available slots capacity to avoid issues like = when a TaskManager is not giving its slots because of some problems = registering the TM; (This means I will lose some performance by not using all the available = capacity) -if a job is failing because of losing a TaskManager (and its slots) the = job will not restart even if available slots are free to use. (for this case the =E2=80=98spare slots=E2=80=99 will not be of help = right; losing a TM means the job will fail, no recovery) Thanks! Best, Ovidiu > On 21 Mar 2016, at 14:15, Robert Metzger wrote: >=20 > Hi Ovidiu, >=20 > right now the scheduler in Flink will not use more slots than = requested. > To avoid issues on recovery, we usually recommend users to have some = spare slots (run job with p=3D15 on a cluster with slots=3D20). I agree = that it would make sense to add a flag which allows a job to grab more = slots if they are available. The problem with that is however, that jobs = can currently not change their parallelism. So if a job fails, it can = not downscale to restart on the remaining slots. > That's why the spare slots approach is currently the only way to go. >=20 > Regards, > Robert >=20 > On Fri, Mar 18, 2016 at 1:30 PM, Ovidiu-Cristian MARCU = > = wrote: > Hi, >=20 > For the situation where a program specify a maximum parallelism (so it = is supposed to use all available task slots) we can have the possibility = that one of the task managers is not registered for various reasons. > In this case the job will fail for not enough free slots to run the = job. >=20 > For me this means the scheduler has a limitation to work by statically = assign tasks to the task slots the job is configured. >=20 > Instead I would like to be able to specify a minimum parallelism of a = job but also the possibility to dynamically use more task slots if = additional task slots can be used. > Another use case will be that if during the execution of a job we lose = one node so some task slots, if the minimum parallelism is still = ensured, the job should recover and continue its execution instead of = just failing. >=20 > Is it possible to make such changes? >=20 > Best, > Ovidiu >=20 --Apple-Mail=_39783C26-9203-41A4-B755-68A4940B005A Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hi Robert,

I am not sure I understand so please confirm if I understand = correctly your suggestions:
- to use less slots = than available slots capacity to avoid issues like when a TaskManager is = not giving its slots because of some problems registering the = TM;
(This means I will lose some performance by not = using all the available capacity)
-if a job is = failing because of losing a TaskManager (and its slots) the job will not = restart even if available slots are free to use.
(for= this case the =E2=80=98spare slots=E2=80=99 will not be of help right; = losing a TM means the job will fail, no recovery)
Thanks!

Best,
Ovidiu


On 21 Mar 2016, at 14:15, Robert Metzger <rmetzger@apache.org>= wrote:

Hi Ovidiu,

right now the scheduler in Flink will not use more slots = than requested.
To avoid issues on recovery, we = usually recommend users to have some spare slots (run job with p=3D15 on = a cluster with slots=3D20). I agree that it would make sense to add a = flag which allows a job to grab more slots if they are available. The = problem with that is however, that jobs can currently not change their = parallelism. So if a job fails, it can not downscale to restart on the = remaining slots.
That's why the spare slots = approach is currently the only way to go.

Regards,
Robert

On Fri, Mar 18, 2016 at 1:30 PM, = Ovidiu-Cristian MARCU <ovidiu-cristian.marcu@inria.fr> wrote:
Hi,

For the situation where a program specify a maximum parallelism (so it = is supposed to use all available task slots) we can have the possibility = that one of the task managers is not registered for various reasons.
In this case the job will fail for not enough free slots to run the = job.

For me this means the scheduler has a limitation to work by statically = assign tasks to the task slots the job is configured.

Instead I would like to be able to specify a minimum parallelism of a = job but also the possibility to dynamically use more task slots if = additional task slots can be used.
Another use case will be that if during the execution of a job we lose = one node so some task slots, if the minimum parallelism is still = ensured, the job should recover and continue its execution instead of = just failing.

Is it possible to make such changes?

Best,
Ovidiu


= --Apple-Mail=_39783C26-9203-41A4-B755-68A4940B005A--