hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bill Sparks <jspa...@cray.com>
Subject Why is my output directory owned by yarn?
Date Tue, 29 Oct 2013 20:19:37 GMT

I have a strange use case and I'm looking for some debugging help.


Use Case:

If I run the hadoop mapped example wordcount program and write the output
to HDFS, the output directory has the correct ownership.

E.g.

hadoop jar 
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.5-alpha.jar
wordcount /user/jdoe/simple/HF.txt /users/jdoe/simple/outtest1

hdfs dfs -ls simple
Found 3 items
drwxr-xr-x - jdoe supergroup 0 2013-10-25 21:26 simple/HF.out
-rw-r--r-- 1 jdoe supergroup 610157 2013-10-25 21:21 simple/HF.txt
drwxr-xr-x - jdoe supergroup 0 2013-10-29 14:50 simple/outtest1

Where as if I write to a global filesystem my output directory is owned by
yarn


E.g.

hadoop jar 
/usr/lib/hadoop-mapreduce/hadoop-mapreduce-examples-2.0.5-alpha.jar
wordcount /user/jdoe/simple/HF.txt file:///scratch/jdoe/outtest1
ls -l /scratch/jdoe
total 8
drwxr-xr-x 2 root root 4096 Oct 28 23:26 logs
drwxr-xr-x 2 yarn yarn 4096 Oct 28 23:23 outtest1



I've looked at the container log files, and saw no errors. The only thing
I can think of, is the user authentication mode is "files:ldap" and the
nodemanager nodes do not have access to the corporate LDAP server so it's
working of local /etc/shadow which does not have my credentials - so it
might just default to "yarn".

I did find the following warning:

2013-10-29 14:58:52,184 INFO
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=jdoe
OPERATION=Container Finished -
Succeeded	TARGET=ContainerImpl	RESULT=SUCCESS	APPID=application_13830201365
44_0005	CONTAINERID=container_1383020136544_0005_01_000001
...
2013-10-29 14:58:53,062 WARN
org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManager
Impl: Trying to stop unknown container
container_1383020136544_0005_01_000001
2013-10-29 14:58:53,062 WARN
org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger:
USER=UnknownUser	IP=10.128.0.17	OPERATION=Stop Container
Request	TARGET=ContainerManagerImpl	RESULT=FAILURE	DESCRIPTION=Trying to
stop unknown 
container!	APPID=application_1383020136544_0005	CONTAINERID=container_13830
20136544_0005_01_000001



 
Thanks,
   John


Mime
View raw message