hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Wilson <paulalexwil...@gmail.com>
Subject YARN Docker launcher failing due to "File is a Directory" for userlog directory
Date Thu, 26 Nov 2015 12:26:59 GMT
Hi all,

First of all I'm running YARN, Hdfs, Oozie within a single docker image,
and I'm using the DockerContainerExecutor within Yarn. This is configured
to use the host's docker daemon to spawn new containers rather than having
a docker-within-docker by passing the docker.sock.
So all daemons are running on the same host alongside spawned YARN
containers. (I'm running Hadoop 2.7.1, Docker 1.7.1).

All jobs fail immediately with, in the stderr of the directory
- /userlogs/application_1448535718658_0001/container_1448535718658_0001_01_000001/stderr
the following message:

log4j:ERROR setFile(null,true) call failed.
java.io.FileNotFoundException:
/usr/local/hadoop/logs/userlogs/application_1448535718658_0001/container_1448535718658_0001_02_000001
(Is a directory)
	at java.io.FileOutputStream.open0(Native Method)
	at java.io.FileOutputStream.open(FileOutputStream.java:270)

When I debug the node manager logs I can see:

exec /bin/bash -c "$JAVA_HOME/bin/java
-Dlog4j.configuration=container-log4j.properties
-Dyarn.app.container.log.dir=/usr/local/hadoop/logs/userlogs/application_1448535718658_0001/container_1448535718658_0001_02_000001
-Dyarn.app.container.log.filesize=0 -Dhadoop.root.logger=INFO,CLA
-Xmx1024m org.apache.hadoop.mapreduce.v2.app.MRAppMaster
1>/usr/local/hadoop/logs/userlogs/application_1448535718658_0001/container_1448535718658_0001_02_000001/stdout
2>/usr/local/hadoop/logs/userlogs/application_1448535718658_0001/container_1448535718658_0001_02_000001/stderr
"

It looks to my inexperienced eye that the 1> and 2> commands are
creating the container_${containerId} directory and then the
yarn.app.container.log.dir is configuring log4j to use the directory
as a log file. Log4j expects that to be a *file* and not a
*directory*. I can't accept this because it would be a clear bug for
everyone, so I'm left thinking it must because of the fact that I'm
sharing the same host between resource manager/node manager and docker
container.

I've tried various things to reconfigure the log directory either from
within Oozie and from yarn-site.xml however nothing seems to work.
When I manually launch everything using the launch-container.sh script
but with a modified yarn.app.container.log.dir path, it appears to
work okay.

So, any help appreciated!

Best Regards,
Paul

Mime
View raw message