hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Badger (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3854) Add localization support for docker images
Date Wed, 10 Oct 2018 20:21:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645496#comment-16645496

Eric Badger commented on YARN-3854:

I definitely prefer the LRU strategy as opposed to {{docker prune}}. It would be nice to have
an "untouchable" list though. Basically images that won't get deleted even if they would qualify
to be deleted using LRU. 

It would also be nice to be able to specify images to delete cluster wide with some sort of
API. Just put the image into the LRU list regardless of where it would have been. 

I'm worried that the image deletion is going to be quite rough and error-prone. Docker containers
sometimes stick around in the {{Exited}} or {{Dead}} states or keep running beyond when they
should. That will cause the delete to perpetually fail since the image is still in use, even
though it isn't _really_ in use, at least not in a useful sense. Do we just keep going around
and around and trying to delete the same images even if we keep failing? 

Since `docker run` implicitly does a `docker pull`, this takes a long time and the container
will be down for a longer time.
This isn't true on the whole. {{docker run}} only does a {{docker pull}} if the image doesn't
exist locally.

> Add localization support for docker images
> ------------------------------------------
>                 Key: YARN-3854
>                 URL: https://issues.apache.org/jira/browse/YARN-3854
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn
>            Reporter: Sidharta Seethana
>            Assignee: Chandni Singh
>            Priority: Major
>              Labels: Docker
>         Attachments: Localization Support For Docker Images.pdf, YARN-3854-branch-2.8.001.patch,
YARN-3854_Localization_support_for_Docker_image_v1.pdf, YARN-3854_Localization_support_for_Docker_image_v2.pdf,
> We need the ability to localize docker images when those images aren't already available
locally. There are various approaches that could be used here with different trade-offs/issues
: image archives on HDFS + docker load ,  docker pull during the localization phase or (automatic)
docker pull during the run/launch phase. 
> We also need the ability to clean-up old/stale, unused images. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message