hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane Kumpf (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3854) Add localization support for docker images
Date Thu, 01 Sep 2016 19:50:22 GMT

    [ https://issues.apache.org/jira/browse/YARN-3854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15456432#comment-15456432

Shane Kumpf commented on YARN-3854:

Today, when credential issues occur during the implicit docker pull, the "image <image
name> not found" error is hidden away in the NM logs. Initially my thought was that failing
fast and exposing the error to the user via the application logs would be sufficient without
credential validation, however, after testing, docker pull has some differing behaviors regarding
various credential issue scenerios. 

If the credentials file is missing or the image does not exist in the registry, docker pull
returns an "image <image name> not found" error (and rc=1). As a result, it will be
difficult to validate if the user supplied a non-existent image or if it's a missing credentials
problem. This deficiency could have been easily addressed via a log message and some documentation

If the credentials exist, but are incorrect (incorrect auth token), it's more challenging
to address as docker falls back to the interactive docker login prompt. Ideally, docker would
provide a command line option to avoid this fall back to make the behavior consistent, and
it may be worth starting that discussion within the docker community towards adding this option.
Otherwise, we would need to try to handle detecting the difference between a long running
docker pull and a docker pull that is waiting for the user to input credentials.

I'll note that, unfortunately, I'm not aware of any options in the docker client CLI to validate
credentials. It may be possible by resorting to checks against the registry REST API, but
ideally, we want to stick with the docker client CLI as we've done so far. I'll look into
a clean way to address validation.

> Add localization support for docker images
> ------------------------------------------
>                 Key: YARN-3854
>                 URL: https://issues.apache.org/jira/browse/YARN-3854
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn
>            Reporter: Sidharta Seethana
>            Assignee: Zhankun Tang
>         Attachments: YARN-3854-branch-2.8.001.patch, YARN-3854_Localization_support_for_Docker_image_v1.pdf,
YARN-3854_Localization_support_for_Docker_image_v2.pdf, YARN-3854_Localization_support_for_Docker_image_v3.pdf
> We need the ability to localize docker images when those images aren't already available
locally. There are various approaches that could be used here with different trade-offs/issues
: image archives on HDFS + docker load ,  docker pull during the localization phase or (automatic)
docker pull during the run/launch phase. 
> We also need the ability to clean-up old/stale, unused images. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message