From user-return-22149-archive-asf-public=cust-asf.ponee.io@flink.apache.org Wed Aug 15 19:53:41 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 3EB87180626 for ; Wed, 15 Aug 2018 19:53:40 +0200 (CEST) Received: (qmail 95966 invoked by uid 500); 15 Aug 2018 17:53:39 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 95956 invoked by uid 99); 15 Aug 2018 17:53:39 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Aug 2018 17:53:39 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id B185AC1589 for ; Wed, 15 Aug 2018 17:53:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.869 X-Spam-Level: * X-Spam-Status: No, score=1.869 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001, T_DKIMWL_WL_MED=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id lx-sXLRY-aCR for ; Wed, 15 Aug 2018 17:53:37 +0000 (UTC) Received: from mail-qt0-f195.google.com (mail-qt0-f195.google.com [209.85.216.195]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 899CC5F3ED for ; Wed, 15 Aug 2018 17:53:36 +0000 (UTC) Received: by mail-qt0-f195.google.com with SMTP id f18-v6so2098082qtp.10 for ; Wed, 15 Aug 2018 10:53:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=EpDSMIfEas4WhLX7r3Dz/u4zEULMrHvNfEXX/Y0/rmc=; b=i8c7g/ITWs64aX2U+7ok+zziWj/VIxlQC2Fp5d+G1ZyKSAjSnCwLP0zs1VC5nTWA9s pv+IUc7B3eog8TI0Ak9KSnZNSGJlDwot5m4yTR5vBJQeysY3SsrnLBZAgkLMf+58+Cth Q3Zi3hHuWhtgt0ZwAaMen5Gk6QEgThjCRCE1dOLtwBQTKVLPJxDb+sAeAV/OI22mltGN RcAaOSjRMeIzAJGu5Wb2hMoz5KPogbm07Og5jYfFiHbZhCCdnEZBJi+g19e/bzoL/zfq o59BoookshsIGWlf5RU2MYjbsXACFrvrP4YwMQXYpmOzRWoHVoOL3UpdjgAgJApidqJ7 yd7w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=EpDSMIfEas4WhLX7r3Dz/u4zEULMrHvNfEXX/Y0/rmc=; b=MC1CtrDx4w5ogCJbkJP2Op+YE+9uIrvneqZN5PlgCetTfEsDksyod5sgnnT1weoTkI 2cp4ftVQLTOO9fg5GuHZEdkfeBm2HDU+6lluXInkGtAt0VDIC0mJPthEfQwSEkQkOGcZ Y39oqIJXo/7KEomkQID48ZKYwmgTexDHJYppkhrtIKdI6Oz9bUsNq3/H4j77xQD9Qm/f da/AKsWc9AZYsLtIg+o/jT8+hawDvLRZqfAdAxf2swksIeRmThf16ME7hXmvByRSezPG GUmYPSLWb50Xy7FIprACyEmpOcGnuOJsdCIM4/PKrzUkEvs8ImIMuVvWEPEChfGuFFwb umBA== X-Gm-Message-State: AOUpUlGRxrDBbyDRSqe9eGk/5sAQgdx3f/lwfGvuTfvYsdUpM6ORx8wD qo/SoeAuBbiLQIQuocKwY/okRase4PHgKTj3hjM= X-Google-Smtp-Source: AA+uWPz6a+OZ35xNW1/e5uFsaQpB1ljlgUrGvpnTFN1edMoS9jkzFXxo1t0Cv8cReKo9Y1EVMdBFJwTRzl4R+BTzeqQ= X-Received: by 2002:ac8:2b45:: with SMTP id 5-v6mr25044764qtv.395.1534355614085; Wed, 15 Aug 2018 10:53:34 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a0c:de0d:0:0:0:0:0 with HTTP; Wed, 15 Aug 2018 10:53:33 -0700 (PDT) In-Reply-To: <5b73fc22.1c69fb81.7f6a1.c0e9@mx.google.com> References: <5b73fc22.1c69fb81.7f6a1.c0e9@mx.google.com> From: shyla deshpande Date: Wed, 15 Aug 2018 10:53:33 -0700 Message-ID: Subject: Re: docker, error NoResourceAvailableException.. To: =?UTF-8?Q?Dominik_Wosi=C5=84ski?= Cc: user Content-Type: multipart/alternative; boundary="00000000000056358a05737d017b" --00000000000056358a05737d017b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thanks Dominik, I will try that. On Wed, Aug 15, 2018 at 3:10 AM, Dominik Wosi=C5=84ski w= rote: > Hey, > The problem is that your command does start Job Manager container, but it > does not start the Task Manager . That is why you have 0 slots. Currently= , > the default *numberOfTaskSlots* is set to the number of CPUs avaialbe on > the machine. > > > So, You generally can to do 2 things: > > > 1) Start Job Manager and 2 Task Managers. If you have Docker Compose > available, you can paste this to your *docker-compose.yml* : > > > > *services: jobmanager: image: *${FLINK_DOCKER_IMAGE_NAME:-flink} > > *expose: *- "6123" > > *ports: *- "8081:8081" > *command: *jobmanager > > *environment: *- JOB_MANAGER_RPC_ADDRESS=3Djobmanager > > > *taskmanager: image: *${FLINK_DOCKER_IMAGE_NAME:-flink} > > *expose: *- "6121" > - "6122" > > *depends_on: *- jobmanager > *command: *taskmanager > > *links: *- "jobmanager:jobmanager" > > *environment: *- JOB_MANAGER_RPC_ADDRESS=3Djobmanager > > > *taskmanager1: image: *${FLINK_DOCKER_IMAGE_NAME:-flink} > > *expose: *- "6190" > - "6120" > > *depends_on: *- jobmanager > *command: *taskmanager > > *links: *- "jobmanager:jobmanager" > > *environment: *- JOB_MANAGER_RPC_ADDRESS=3Djobmanager > > > > This will give you 1 Job Manager and 2 Task Managers with one task slot > each, so 2 Task slots in general. > > 2) You can deploy 1 Job Manager and 1 Task Manager.Then you need to modif= y > *flink-conf.yml* and set the following setting : > > *taskmanager.numberOfTaskSlots: *2 > > > This will give you 2 Task Slots with only 1 Task Manager. But you will > need to somehow override config in the container, possibly using : > https://docs.docker.com/storage/volumes/ > > Regards, > Dominik. > > *Od: *shyla deshpande > *Wys=C5=82ano: *=C5=9Broda, 15 sierpnia 2018 07:23 > *Do: *user > *Temat: *docker, error NoResourceAvailableException.. > > > > Hello all, > > > > Trying to use docker as a single node flink cluster. > > > > docker run --name flink_local -p 8081:8081 -t flink local > > > > I submited a job to the cluster using the Web UI. The job failed. I see > this error message in the docker logs. > > > > org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableExceptio= n: > Could not allocate all requires slots within timeout of 300000 ms. Slots > required: 2, slots allocated: 0 > > > > The Web UI, shows 0 taskmanagers and 0 task slots on the Flink dashboard. > > How do I start the docker with 2 Task slots? > > > > Appreciate any help. > > > > Thanks > > > --00000000000056358a05737d017b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Thanks=C2=A0Do= minik, I will try that.

On Wed, Aug 15, 2018 at 3:10 AM, Dominik Wosi=C5=84ski <= span dir=3D"ltr"><= wossyn@gmail.com> wrote:

Hey,
The problem is that y= our command does start Job Manager container, but it does not start the Tas= k Manager . That is why you have 0 slots. Currently, the default numberO= fTaskSlots is set to the number of CPUs avaialbe on the machine.


So, You generally can to do 2 things:


1) Start Job Manager and 2 Task Managers. If you have Do= cker Compose available, you can paste this to your docker-compose.yml : 

=
services:
=C2=A0 jobmanager:
=C2=A0=C2=A0=C2=A0 image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
=C2=A0=
=C2=A0=C2=A0 expose:
=C2=A0=C2=
=A0=C2=A0=C2=A0=C2=A0 - "6123"
=C2=A0=C2=A0=C2=A0 ports:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - "8081:8081"
=C2=A0=C2=A0=C2=A0 command: jobmanager
=
=C2=A0=C2=A0=C2=A0 environment:
=
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - J=
OB_MANAGER_RPC_ADDRESS=3Djobmanager

=C2=A0 taskmanager:
=C2=A0=C2=A0=C2=A0 image: ${FLINK_DOCKER_IMAGE_NAME:-flink}
=C2=
=A0=C2=A0=C2=A0 expose:
=C2=A0=
=C2=A0=C2=A0=C2=A0=C2=A0 - =
"6121"
=C2=A0=C2=A0=C2=A0=C2=A0=
=C2=A0 - "6122"
=C2=A0=C2=A0=C2=A0 depends_on:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - jobmanager
=C2=A0=C2=A0=C2=A0 command: t=
askmanager
=C2=A0=C2=A0=C2=A0 li=
nks:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - "jobmanager:jobmanager&quo=
t;
=C2=A0=C2=A0=C2=A0 environmen=
t:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - JOB_MANAGER_RPC_ADDRESS=3Djobmanager

=C2=A0 taskmanager1:
=C2=A0=C2=A0=C2=A0 image: ${FLINK_DOCKER_IMAGE_NAME:-flink}=C2=A0=C2=A0=C2=A0 expose:
=C2=
=A0=C2=A0=C2=A0=C2=A0=C2=A0 - "6190"
=C2=A0=C2=A0=C2=A0=C2=
=A0=C2=A0 - "6120"
=C2=A0=C2=A0=C2=A0 depends_on:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - jobmanager
=C2=A0=C2=A0=C2=A0 command: taskmanager
=C2=A0=C2=A0=C2=A0 links:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - "jobmanager:jobmanager&=
quot;
=C2=A0=C2=A0=C2=A0 environ=
ment:
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 - JOB_MANAGER_RPC_ADDRESS=3Djobmanager
=C2=A0
This=
 will give you 1 Job Manager and 2 Task Managers with one task slot each, s=
o 2 Task slots in general.

2) You can dep=
loy 1 Job Manager and 1 Task Manager.Then you need to modify flink-conf.=
yml and set the following setting : 

taskmanager.numbe=
rOfTaskSlots: 2

This will give you 2 Task Slots with onl=
y 1 Task Manager. But you will need to somehow override config in the conta=
iner, possibly using : https://docs.docker.com/storage/volumes/
Regards,
Dominik.
Od: shyla deshpande
Wys=C5=82ano: =
=C5=9Broda, 15 sierpnia 2018 07:23
Do: user
Temat: docker, erro=
r NoResourceAvailableException..
=C2=A0
Hel=
lo all,
=C2=A0
=

Trying to use docker as a single node flink clu=
ster.
=C2=A0
docker run --name flink_local -p 8081:8081 -t fli=
nk local
=C2=A0
I submited a job to the cluster using the Web =
UI. The job failed. I see this error message in the docker logs.
<=
div>
=C2=A0
org.apache.flink.runtime.jobmanager.scheduler.NoResourc=
eAvailableException: Could not allocate all requires slots within timeout o=
f 300000 ms. Slots required: 2, slots allocated: 0
=C2=A0
Th=
e Web UI, shows 0 taskmanagers and 0 task slots on the Flink dashboard.
=

How do I start the docker with 2 Task slo=
ts?
=C2=A0
Appreciate any help.
=C2=A0
Thanks
=C2=A0

--00000000000056358a05737d017b--