hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Templeton (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3854) Add localization support for docker images
Date Wed, 20 Jul 2016 18:33:20 GMT

    [ https://issues.apache.org/jira/browse/YARN-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15386391#comment-15386391
] 

Daniel Templeton commented on YARN-3854:
----------------------------------------

[~tangzhankun], thanks for the doc and patch.

It appears that this patch focuses only on using the distributed cache as the vehicle for
sharing Docker images.  While that works, it would be more desirable to have a Docker pull
happen from a Docker repo.  Otherwise the user has to pull the image out of the repo to stuff
it into HDFS, tripling the total amount of data moved over the network (repo->client,client->HDFS,HDFS->NM
rather than repo->NM).  The image files may be very large, making the extra data movement
and storage a potential issue.  This approach also creates the potential for image version
mismatches and forces the user to manage the image layers.  For these reasons, I think it
would make more sense to have the localization process execute a Docker pull for the image
on the target host instead.

When considering localization through a Docker pull command, you should also consider how
to deal with secure registries.

> Add localization support for docker images
> ------------------------------------------
>
>                 Key: YARN-3854
>                 URL: https://issues.apache.org/jira/browse/YARN-3854
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn
>            Reporter: Sidharta Seethana
>            Assignee: Zhankun Tang
>         Attachments: YARN-3854-branch-2.8.001.patch, YARN-3854_Localization_support_for_Docker_image_v1.pdf,
YARN-3854_Localization_support_for_Docker_image_v2.pdf
>
>
> We need the ability to localize images from HDFS and load them for use when launching
docker containers. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message