hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venu Gopala Rao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2297) All map reduce tasks are failing if we give invalid path jar file for Job
Date Tue, 24 May 2011 06:35:47 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13038414#comment-13038414
] 

Venu Gopala Rao commented on MAPREDUCE-2297:
--------------------------------------------

Hi Todd, I feel the fix will be useful. Let us just consider the case of Java, even one of
the paths mentioned in classpath is invalid, Java will execute. As you mentioned  , if it
is a typo error and a required jar is missing, the program may get ClassNotFoundExceptions.

Since hive.aux.jars.path is a system level configuration, we should not fail all jobs. But
some jobs may get ClassNotFoundExceptions if a required jar is missing. 

What you say?

> All map reduce tasks are failing if we give invalid path jar file for Job
> -------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-2297
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2297
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.20.2
>            Reporter: Devaraj K
>            Assignee: Devaraj K
>            Priority: Minor
>             Fix For: 0.20.4
>
>         Attachments: MAPREDUCE-2297.patch
>
>
> This can be reproduced by giving the invalid jar file for the Job or it can be reproduced
from hive.
> In hive-default.xml
> <property>
> <name>hive.aux.jars.path</name>
> <value></value>
> <description>Provided for adding auxillaryjarsPath</description>
> </property>
> If we configure an invalid path for jar file, It is making all map reduce tasks to fail
even those jobs are not depending on this jar file and it is giving the below exception.
> {code:xml} 
> hive> select * from a join b on(a.b=b.c);
> Total MapReduce jobs = 1
> Launching Job 1 out of 1
> Number of reduce tasks not specified. Estimated from input data size: 1
> In order to change the average load for a reducer (in bytes):
> set hive.exec.reducers.bytes.per.reducer=<number>
> In order to limit the maximum number of reducers:
> set hive.exec.reducers.max=<number>
> In order to set a constant number of reducers:
> set mapred.reduce.tasks=<number>
> java.io.FileNotFoundException: File does not exist: /user/root/grade.jar
> at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:495)
> at org.apache.hadoop.filecache.DistributedCache.getTimestamp(DistributedCache.java:509)
> at org.apache.hadoop.mapred.JobClient.configureCommandLineOptions(JobClient.java:651)
> at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:783)
> at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:752)
> at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:698)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:107)
> at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:64)
> {code} 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message