flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Esteban Serrano <st96d...@gmail.com>
Subject Re: docker, error NoResourceAvailableException..
Date Wed, 15 Aug 2018 18:27:46 GMT
You can also instead of defining 2 services (taskmanager and taskmanager1),
set the scale parameter on taskmanager to the number of desired slots.
Something like this:

taskmanager:
  image: "${FLINK_DOCKER_IMAGE:-flink:1.5.2}"
  scale: 2
  expose:
    - "6121"
    - "6122"
    - "8081"


On Wed, Aug 15, 2018 at 1:53 PM shyla deshpande <deshpandeshyla@gmail.com>
wrote:

> Thanks Dominik, I will try that.
>
> On Wed, Aug 15, 2018 at 3:10 AM, Dominik Wosiński <wossyn@gmail.com>
> wrote:
>
>> Hey,
>> The problem is that your command does start Job Manager container, but it
>> does not start the Task Manager . That is why you have 0 slots. Currently,
>> the default *numberOfTaskSlots* is set to the number of CPUs avaialbe on
>> the machine.
>>
>>
>> So, You generally can to do 2 things:
>>
>>
>> 1) Start Job Manager and 2 Task Managers. If you have Docker Compose
>> available, you can paste this to your *docker-compose.yml* :
>>
>>
>>
>> *services:  jobmanager:    image: *${FLINK_DOCKER_IMAGE_NAME:-flink}
>>
>> *expose:      *- "6123"
>>
>> *ports:      *- "8081:8081"
>>     *command: *jobmanager
>>
>> *environment:      *- JOB_MANAGER_RPC_ADDRESS=jobmanager
>>
>>
>> *taskmanager:    image: *${FLINK_DOCKER_IMAGE_NAME:-flink}
>>
>> *expose:      *- "6121"
>>       - "6122"
>>
>> *depends_on:      *- jobmanager
>>     *command: *taskmanager
>>
>> *links:      *- "jobmanager:jobmanager"
>>
>> *environment:      *- JOB_MANAGER_RPC_ADDRESS=jobmanager
>>
>>
>> *taskmanager1:    image: *${FLINK_DOCKER_IMAGE_NAME:-flink}
>>
>> *expose:      *- "6190"
>>       - "6120"
>>
>> *depends_on:      *- jobmanager
>>     *command: *taskmanager
>>
>> *links:      *- "jobmanager:jobmanager"
>>
>> *environment:      *- JOB_MANAGER_RPC_ADDRESS=jobmanager
>>
>>
>>
>> This will give you 1 Job Manager and 2 Task Managers with one task slot
>> each, so 2 Task slots in general.
>>
>> 2) You can deploy 1 Job Manager and 1 Task Manager.Then you need to
>> modify *flink-conf.yml* and set the following setting :
>>
>> *taskmanager.numberOfTaskSlots: *2
>>
>>
>> This will give you 2 Task Slots with only 1 Task Manager. But you will
>> need to somehow override config in the container, possibly using :
>> https://docs.docker.com/storage/volumes/
>>
>> Regards,
>> Dominik.
>>
>> *Od: *shyla deshpande <deshpandeshyla@gmail.com>
>> *Wysłano: *środa, 15 sierpnia 2018 07:23
>> *Do: *user <user@flink.apache.org>
>> *Temat: *docker, error NoResourceAvailableException..
>>
>>
>>
>> Hello all,
>>
>>
>>
>> Trying to use docker as a single node flink cluster.
>>
>>
>>
>> docker run --name flink_local -p 8081:8081 -t flink local
>>
>>
>>
>> I submited a job to the cluster using the Web UI. The job failed. I see
>> this error message in the docker logs.
>>
>>
>>
>> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
>> Could not allocate all requires slots within timeout of 300000 ms. Slots
>> required: 2, slots allocated: 0
>>
>>
>>
>> The Web UI, shows 0 taskmanagers and 0 task slots on the Flink dashboard.
>>
>> How do I start the docker with 2 Task slots?
>>
>>
>>
>> Appreciate any help.
>>
>>
>>
>> Thanks
>>
>>
>>
>
>

Mime
View raw message