hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eric Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7430) User and Group mapping are incorrect in docker container
Date Mon, 13 Nov 2017 22:00:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-7430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16250312#comment-16250312
] 

Eric Yang commented on YARN-7430:
---------------------------------

[~ebadger] {quote}
I could be wrong, but I don't think Shane Kumpf's question is whether this happens in docker,
but whether this is possible to happen in hadoop. i.e. can the above docker run (or similar)
ever actually be created by the NM starting up a docker container.
{quote}

Yes, it's possible in Hadoop.  Here is the container log for application master:
{code}
2017-11-13 21:24:57,440 [pool-5-thread-4] INFO  registry.YarnRegistryViewForProviders - [COMPINSTANCE
kafkabroker-0 : container_1510599241403_0006_01_000002]: Deleting registry path /users/1234/services/yarn-service/amp/components/ctr-1510599241403-0006-01-000002
2017-11-13 21:24:57,441 [CompInstance dispatcher] INFO  instance.ComponentInstance - [COMPINSTANCE
kafkabroker-0] Transitioned from STARTED to INIT on STOP event
2017-11-13 21:24:59,449 [AMRM Callback Handler Thread] INFO  service.ServiceScheduler - 1
containers allocated. 
2017-11-13 21:24:59,450 [AMRM Callback Handler Thread] INFO  service.ServiceScheduler - [COMPONENT
kafkabroker]: 1 outstanding container requests.
2017-11-13 21:24:59,450 [AMRM Callback Handler Thread] INFO  service.ServiceScheduler - [COMPONENT
kafkabroker]: removing one container request.
2017-11-13 21:24:59,450 [Component  dispatcher] INFO  component.Component - [COMPONENT kafkabroker]:
container_1510599241403_0006_01_000003 allocated, num pending component instances reduced
to 0
2017-11-13 21:24:59,450 [Component  dispatcher] INFO  component.Component - [COMPONENT kafkabroker]:
Assigned container_1510599241403_0006_01_000003 to component instance kafkabroker-0 and launch
on host eyang-5.openstacklocal:34611 
2017-11-13 21:24:59,454 [pool-6-thread-1] INFO  provider.ProviderUtils - Component instance
conf dir already exists: hdfs://eyang-1:9000/user/1234/.yarn/services/amp/components/kafkabroker/kafkabroker-0
2017-11-13 21:24:59,466 [pool-6-thread-1] INFO  provider.ProviderUtils - Add config file for
localization: conf/server.properties -> /user/1234/.yarn/services/amp/components/kafkabroker/kafkabroker-0/server.properties,
dest mount path: /etc/kafka/conf/server.properties
2017-11-13 21:24:59,467 [pool-6-thread-1] INFO  containerlaunch.AbstractLauncher - yarn docker
env var has been set {LANGUAGE=en_US.UTF-8, HADOOP_USER_NAME=1234, YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_HOSTNAME=kafkabroker-0.amp.1234,
WORK_DIR=$PWD, LC_ALL=en_US.UTF-8, YARN_CONTAINER_RUNTIME_DOCKER_LOCAL_RESOURCE_MOUNTS=conf/server.properties:/etc/kafka/conf/server.properties,
YARN_CONTAINER_RUNTIME_TYPE=docker, YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=registry.eng.hortonworks.com/hwx-assemblies/kafka:0.10.1,
LANG=en_US.UTF-8, YARN_CONTAINER_RUNTIME_DOCKER_CONTAINER_NETWORK=bridge, LOG_DIR=<LOG_DIR>,
YARN_CONTAINER_RUNTIME_DOCKER_RUN_PRIVILEGED_CONTAINER=false}
2017-11-13 21:24:59,468 [org.apache.hadoop.yarn.client.api.async.impl.NMClientAsyncImpl #1]
INFO  impl.NMClientAsyncImpl - Processing Event EventType: START_CONTAINER for Container container_1510599241403_0006_01_000003
2017-11-13 21:24:59,487 [CompInstance dispatcher] INFO  instance.ComponentInstance - [COMPINSTANCE
kafkabroker-0 : container_1510599241403_0006_01_000003] Transitioned from INIT to STARTED
on START event
2017-11-13 21:25:00,458 [Component  dispatcher] INFO  component.Component - [COMPONENT kafkabroker]:
container_1510599241403_0006_01_000003 completed, num pending comp instances increased to
1.
{code}

And screenshot attached in the attachement section.

> User and Group mapping are incorrect in docker container
> --------------------------------------------------------
>
>                 Key: YARN-7430
>                 URL: https://issues.apache.org/jira/browse/YARN-7430
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: security, yarn
>    Affects Versions: 2.9.0, 3.0.0
>            Reporter: Eric Yang
>            Assignee: Eric Yang
>            Priority: Blocker
>         Attachments: YARN-7430.001.patch, YARN-7430.png
>
>
> In YARN-4266, the recommendation was to use -u [uid]:[gid] numeric values to enforce
user and group for the running user.  In YARN-6623, this translated to --user=test --group-add=group1.
 The code no longer enforce group correctly for launched process.  
> In addition, the implementation in YARN-6623 requires the user and group information
to exist in container to translate username and group to uid/gid.  For users on LDAP, there
is no good way to populate container with user and group information. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message