hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Groschupf ...@media-style.com>
Subject crashing in job initializing block other jobs
Date Thu, 09 Mar 2006 19:58:16 GMT
Hi Developers,

I run in situations where it a failed job blocked all new jobs.

in may case I had a exception  in the InputFormat. E.g. no data in  
the defined input folder.
This clauses a exception in the JobInProgress#initTasks.
In this case the fields 'maps' and 'reduces' of JobInProgress are  
still null.

After that the jobtracker tries to kill the job but this fails also  
with a NPE, so the job is never killed and runs for ever and blocks  
other jobs in my case.

I see two ways to solve that:
1.) check if maps and reduces are null in the JobInProgress object ala:
if(maps != null){
               for (int i = 0; i < maps.length; i++) {
                   maps[i].kill();
               }
similar for the reduces.

2.) 	add a exception to taskInProgress#kill and JobInProgress#kill  
and handle this in the Jobtracker.

3.) stop the JobInitThread in case the initialization fails.


Any thoughts? Should I submit a patch?

Find the exceptions below.
Greetings,
Stefan



java.io.IOException: No input directories specified in:  
Configuration: defaults: hadoop-default.xml , mapred-default.xml , / 
tmp/hadoop/mapred/local/jobTracker/job_bwq6vj.xmlfinal: hadoop-site.xml
	at org.apache.hadoop.mapred.InputFormatBase.listFiles 
(InputFormatBase.java:84)
	at org.apache.hadoop.mapred.InputFormatBase.getSplits 
(InputFormatBase.java:94)
	at org.apache.hadoop.mapred.JobInProgress.initTasks 
(JobInProgress.java:127)
	at org.apache.hadoop.mapred.JobTracker$JobInitThread.run 
(JobTracker.java:208)
	at java.lang.Thread.run(Thread.java:613)

Exception in thread "Thread-25" java.lang.NullPointerException
	at org.apache.hadoop.mapred.JobInProgress.kill(JobInProgress.java:437)
	at org.apache.hadoop.mapred.JobTracker$JobInitThread.run 
(JobTracker.java:212)
	at java.lang.Thread.run(Thread.java:613)



---------------------------------------------------------------
company:        http://www.media-style.com
forum:        http://www.text-mining.org
blog:            http://www.find23.net



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message