hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane Kumpf (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5428) Allow for specifying the docker client configuration directory
Date Fri, 03 Nov 2017 12:21:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-5428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16237533#comment-16237533
] 

Shane Kumpf commented on YARN-5428:
-----------------------------------

Thanks for the feedback @vinodkv! After a long hiatus, I've spent a bit of time looking at
the approach you laid out and think it makes sense, despite the cons. 

The approach aligns with the handling of existing credentials and also helps eliminate the
ordering challenges around image localization, since the tokens will already be available
(client config must be localized before docker image pull). 

The major con being that each client will need to add support for a new CLI option/endpoint
so that the user can supply the docker client config via the client. You mention that most
frameworks already have some level of distributed cache support, however, Credentials and
LocalResources aren't handled the same AFAICT, so a new option needs to be added. I also don't
believe it would be appropriate to add this to GenericOptionsParser, given it only applies
to applications. 

Based on the above, my thought is to provide support in distributed shell as part of this
patch and a follow on for support in YARN Native Services. Enabling this feature for additional
clients could follow based on need. Does that sound reasonable?

> Allow for specifying the docker client configuration directory
> --------------------------------------------------------------
>
>                 Key: YARN-5428
>                 URL: https://issues.apache.org/jira/browse/YARN-5428
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: yarn
>            Reporter: Shane Kumpf
>            Assignee: Shane Kumpf
>            Priority: Major
>              Labels: oct16-medium
>         Attachments: YARN-5428.001.patch, YARN-5428.002.patch, YARN-5428.003.patch, YARN-5428.004.patch,
YARN-5428Allowforspecifyingthedockerclientconfigurationdirectory.pdf
>
>
> The docker client allows for specifying a configuration directory that contains the docker
client's configuration. It is common to store "docker login" credentials in this config, to
avoid the need to docker login on each cluster member. 
> By default the docker client config is $HOME/.docker/config.json on Linux. However, this
does not work with the current container executor user switching and it may also be desirable
to centralize this configuration beyond the single user's home directory.
> Note that the command line arg is for the configuration directory NOT the configuration
file.
> This change will be needed to allow YARN to automatically pull images at localization
time or within container executor.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message