hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shane Kumpf (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-8623) Update Docker examples to use image which exists
Date Mon, 20 Aug 2018 22:33:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-8623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16586601#comment-16586601
] 

Shane Kumpf commented on YARN-8623:
-----------------------------------

I was able to run the below MR pi job with the modified _apache/hadoop-runner_ image, after
a quick hack to the sudo rules.
{code:java}
YARN_EXAMPLES_JAR=$HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar
IMAGE_ID="local/hadoop-runner-new:latest"
MOUNTS="/usr/local/hadoop:/usr/local/hadoop:ro,/etc/hadoop/conf:/etc/hadoop/conf:ro,/etc/passwd:/etc/passwd:ro,/etc/group:/etc/group:ro"

yarn jar $YARN_EXAMPLES_JAR pi \
-Dmapreduce.map.env.YARN_CONTAINER_RUNTIME_TYPE=docker \
-Dmapreduce.map.env.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=$MOUNTS \
-Dmapreduce.map.env.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$IMAGE_ID \
-Dmapreduce.reduce.env.YARN_CONTAINER_RUNTIME_TYPE=docker \
-Dmapreduce.reduce.env.YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS=$MOUNTS \
-Dmapreduce.reduce.env.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$IMAGE_ID 1 40000{code}
Hadoop bits were installed to {{/usr/local/hadoop}} on the host. Hadoop config in {{/etc/hadoop/conf}} on
the host. The appropriate mounts were added to {{docker.allowed.ro-mounts}} and the image
prefix to {{docker.trusted.registries}} in {{container-executor.cfg}}

The above assumes the use of {{/etc/passwd}} and {{/etc/group}} for propagating the user
and group into the container. We should point to the other ways of [managing user propagation|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md#user-management-in-docker-container]
as part of this example documentation.

> Update Docker examples to use image which exists
> ------------------------------------------------
>
>                 Key: YARN-8623
>                 URL: https://issues.apache.org/jira/browse/YARN-8623
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Craig Condit
>            Priority: Minor
>              Labels: Docker
>
> The example Docker image given in the documentation (images/hadoop-docker:latest) does
not exist. We could change images/hadoop-docker:latest to apache/hadoop-runner:latest, which
does exist. We'd need to do a quick sanity test to see if the image works with YARN.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message