hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Groschupf ...@media-style.com>
Subject Re: crashing in job initializing block other jobs
Date Thu, 09 Mar 2006 20:02:55 GMT
May be also a solution:
4.) initialize the maps and reduces fields with an array length 0.


Am 09.03.2006 um 20:58 schrieb Stefan Groschupf:

> Hi Developers,
>
> I run in situations where it a failed job blocked all new jobs.
>
> in may case I had a exception  in the InputFormat. E.g. no data in  
> the defined input folder.
> This clauses a exception in the JobInProgress#initTasks.
> In this case the fields 'maps' and 'reduces' of JobInProgress are  
> still null.
>
> After that the jobtracker tries to kill the job but this fails also  
> with a NPE, so the job is never killed and runs for ever and blocks  
> other jobs in my case.
>
> I see two ways to solve that:
> 1.) check if maps and reduces are null in the JobInProgress object  
> ala:
> if(maps != null){
>               for (int i = 0; i < maps.length; i++) {
>                   maps[i].kill();
>               }
> similar for the reduces.
>
> 2.) 	add a exception to taskInProgress#kill and JobInProgress#kill  
> and handle this in the Jobtracker.
>
> 3.) stop the JobInitThread in case the initialization fails.
>
>
> Any thoughts? Should I submit a patch?
>
> Find the exceptions below.
> Greetings,
> Stefan
>
>
>
> java.io.IOException: No input directories specified in:  
> Configuration: defaults: hadoop-default.xml , mapred-default.xml , / 
> tmp/hadoop/mapred/local/jobTracker/job_bwq6vj.xmlfinal: hadoop- 
> site.xml
> 	at org.apache.hadoop.mapred.InputFormatBase.listFiles 
> (InputFormatBase.java:84)
> 	at org.apache.hadoop.mapred.InputFormatBase.getSplits 
> (InputFormatBase.java:94)
> 	at org.apache.hadoop.mapred.JobInProgress.initTasks 
> (JobInProgress.java:127)
> 	at org.apache.hadoop.mapred.JobTracker$JobInitThread.run 
> (JobTracker.java:208)
> 	at java.lang.Thread.run(Thread.java:613)
>
> Exception in thread "Thread-25" java.lang.NullPointerException
> 	at org.apache.hadoop.mapred.JobInProgress.kill(JobInProgress.java: 
> 437)
> 	at org.apache.hadoop.mapred.JobTracker$JobInitThread.run 
> (JobTracker.java:212)
> 	at java.lang.Thread.run(Thread.java:613)
>
>
>
> ---------------------------------------------------------------
> company:        http://www.media-style.com
> forum:        http://www.text-mining.org
> blog:            http://www.find23.net
>
>

---------------------------------------------------------------
company:        http://www.media-style.com
forum:        http://www.text-mining.org
blog:            http://www.find23.net



Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message