hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Pierre-Francois Laquerre <pierre.franc...@nec-labs.com>
Subject incorrect staging area path in 2.0.5
Date Thu, 01 Aug 2013 14:57:26 GMT
I recently updated from 1.0.4 to 2.0.5. Since then, streaming jobs have 
been failing to launch due to what seems like an incorrect staging path:

# /opt/hadoop2/bin/hadoop jar 
/opt/hadoop2/share/hadoop/tools/lib/hadoop-streaming-2.0.5-alpha.jar 
-input foo -output bar -mapper baz -reducer foobar
13/08/01 10:43:50 WARN util.NativeCodeLoader: Unable to load 
native-hadoop library for your platform... using builtin-java classes 
where applicable
13/08/01 10:43:50 WARN conf.Configuration: session.id is deprecated. 
Instead, use dfs.metrics.session-id
13/08/01 10:43:50 INFO jvm.JvmMetrics: Initializing JVM Metrics with 
processName=JobTracker, sessionId=
13/08/01 10:43:50 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics 
with processName=JobTracker, sessionId= - already initialized
13/08/01 10:43:50 INFO mapreduce.JobSubmitter: Cleaning up the staging 
area *file:/user/myuser1544460269/.staging/job_local1544460269_0001*
13/08/01 10:43:50 ERROR security.UserGroupInformation: 
PriviledgedActionException as:myuser (auth:SIMPLE) 
cause:org.apache.hadoop.util.Shell$ExitCodeException: chmod: cannot 
access `*/user/myuser1544460269/.staging/job_local1544460269_0001*': No 
such file or directory
13/08/01 10:43:50 ERROR streaming.StreamJob: Error Launching job : 
chmod: cannot access 
`/user/myuser1544460269/.staging/job_local1544460269_0001': No such file 
or directory
Streaming Command Failed!

This is for a job launched as "myuser". Given that 
mapreduce.jobtracker.staging.root.dir is set to /user, I would expect 
the staging area to be in /user/myuser/.staging/job_local$jobid. 
Instead, it is in /user/myuser$jobid/.staging/job_local$jobid, which 
fails since /user/myuser$jobid/ doesn't exist. Has the way 
staging.root.dir is used changed in 2.x?

Thank you,

Pierre


Mime
View raw message