hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen He (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3289) Docker images should be downloaded during localization
Date Wed, 04 Mar 2015 20:11:38 GMT

    [ https://issues.apache.org/jira/browse/YARN-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347491#comment-14347491

Chen He commented on YARN-3289:

Thank [~jlowe] for the comments. IMHO, we can move the docker image localization into a preparation
If we are using DCE for running applications. For example, we have 10 task in a job, we create
extra 1 "tasks" for each real task.
I mean, start a extra dummy task that can heartbeat and do the image downloading work. Once
it is done, the real task can start to run. 

The benefit is that we can control the placement of those dummy tasks and achieve "data locality"
for docker image localization. 
For example:
   we have node1 which has already downloaded the docker image and AM starts to run on it.
If possible, RM scheduler should put other dummy and real task on this node since node1 has
already has the image. Comparing with job input data (a block? maybe), the docker image "locality"
(more than 10 min to download a image, it will be more than 2GB) may be more important. 

> Docker images should be downloaded during localization
> ------------------------------------------------------
>                 Key: YARN-3289
>                 URL: https://issues.apache.org/jira/browse/YARN-3289
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Ravi Prakash
> We currently call docker run on images while launching containers. If the image size
if sufficiently big, the task will timeout. We should download the image we want to run during
localization (if possible) to prevent this

This message was sent by Atlassian JIRA

View raw message