hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeffrey Naisbitt (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2716) MR279: MRReliabilityTest job fails because of missing job-file.
Date Fri, 29 Jul 2011 21:55:09 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13073038#comment-13073038
] 

Jeffrey Naisbitt commented on MAPREDUCE-2716:
---------------------------------------------

Thanks for the review, Vinod.  That makes sense.

Another question/concern though: I see that in JobSubmitter.java (submitJobInternal), it originally
generates the "submitJobDir" based on a call to JobSubmissionFiles.getStagingDir & the
jobId.  (where getStagingDir ultimately goes off to YARNRunner.getStagingAreaDir(), which
goes to the ResourceMgrDelegate & then MRApps).  Anyway,  that staging area directory
seems to include the user name and ".staging" as well as the value from the "yarn.apps.stagingDir".
Is that the value we should be using instead of just getting the setting from conf?  If so,
maybe it would make sense to just pass the actual staging directory in instead of the actual
Configuration object.


> MR279: MRReliabilityTest job fails because of missing job-file.
> ---------------------------------------------------------------
>
>                 Key: MAPREDUCE-2716
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2716
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Jeffrey Naisbitt
>            Assignee: Jeffrey Naisbitt
>             Fix For: 0.23.0
>
>         Attachments: MAPREDUCE-2716-v2.patch, MAPREDUCE-2716-v3.patch, MAPREDUCE-2716.patch
>
>
> The ApplicationReport should have the jobFile (e.g. hdfs://localhost:9000/tmp/hadoop-<USER>/mapred/staging/<USER>/.staging/job_201107121640_0001/job.xml)
> Without it, jobs such as MRReliabilityTest fail with the following error (caused by the
fact that jobFile is hardcoded to "" in TypeConverter.java):
> e.g. java.lang.IllegalArgumentException: Can not create a Path from an empty string
>         at org.apache.hadoop.fs.Path.checkPathArg(Path.java:88)
>         at org.apache.hadoop.fs.Path.<init>(Path.java:96)
>         at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:445)
>         at org.apache.hadoop.mapreduce.Cluster.getJobs(Cluster.java:104)
>         at org.apache.hadoop.mapreduce.Cluster.getAllJobs(Cluster.java:218)
>         at org.apache.hadoop.mapred.JobClient.getAllJobs(JobClient.java:757)
>         at org.apache.hadoop.mapred.JobClient.jobsToComplete(JobClient.java:741)
>         at org.apache.hadoop.mapred.ReliabilityTest.runTest(ReliabilityTest.java:219)
>         at org.apache.hadoop.mapred.ReliabilityTest.runSleepJobTest(ReliabilityTest.java:133)
>         at org.apache.hadoop.mapred.ReliabilityTest.run(ReliabilityTest.java:116)
>         at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:69)
>         at org.apache.hadoop.mapred.ReliabilityTest.main(ReliabilityTest.java:504)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
>         at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
>         at org.apache.hadoop.test.MapredTestDriver.run(MapredTestDriver.java:111)
>         at org.apache.hadoop.test.MapredTestDriver.main(MapredTestDriver.java:118)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.apache.hadoop.util.RunJar.main(RunJar.java:192)

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message