hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ferreira, Herve (NSN - PT/Amadora)" <herve.ferre...@nsn.com>
Subject Reduce hangs, FileNotFoundException
Date Mon, 28 Dec 2009 15:58:57 GMT
 
Hi, 


I'm really frustrated because I've already lost some days trying to 
deploy hadoop but it doesn't work. 


If I deploy in a single cluster all things work ok (the mapreduce 
example as well the deployment). 


However when I try to install hadoop in a cluster the problems appear. 
The configurations are the one that you shown in your blog. 


After I start the cluster I check the logs and no exceptions are 
thown. 
When I start the hadoop-example wordcount the map task is executed 
without problems ( INFO mapred.JobClient: map 100% reduce 0%) but then 
stops where and in the console shows "Error reading task 
outputhttp://master:50060/tasklog?plaintext=true&taskid=attempt_20091228132... <http://www.google.com/url?sa=D&q=http://master:50060/tasklog%3Fplaintext%3Dtrue%26taskid%3Dattempt_200912281326_0001_m_000000_0%26filter%3Dstdout%E2%80%9D&usg=AFQjCNGexaEsrP7sB9Slpcjrxi-wQNWaMA>
 


If I check the log files I can see the following exceptions: 
In the /userLogs/attemptX 
2009-12-28 13:28:36,035 WARN org.apache.hadoop.mapred.ReduceTask: 
java.io.FileNotFoundException: 
http://master:50060/mapOutput?job=job_200912281326_0001&map=attempt_2... <http://www.google.com/url?sa=D&q=http://master:50060/mapOutput%3Fjob%3Djob_200912281326_0001%26map%3Dattempt_200912281326_0001_m_000000_0%26reduce%3D0&usg=AFQjCNHUolZY6jZh4toz_Zsx1tISJS8OUA>
 
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method) 
at sun.reflect.NativeConstructorAccessorImpl.newInstance 
(NativeConstructorAccessorImpl.java:39) 
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance 
(DelegatingConstructorAccessorImpl.java:27) 
at java.lang.reflect.Constructor.newInstance(Constructor.java:513) 
at sun.net.www.protocol.http.HttpURLConnection$6.run <http://www.google.com/url?sa=D&q=www.protocol.http.HttpURLConnection%246.run&usg=AFQjCNF3SFlsqNm7hY5we-D3sXnMDeM1cA>
 
(HttpURLConnection.java:1345) 
at java.security.AccessController.doPrivileged(Native Method) 
at sun.net.www.protocol.http.HttpURLConnection.getChainedException <http://www.google.com/url?sa=D&q=www.protocol.http.HttpURLConnection.getChainedException&usg=AFQjCNFWiaqz2t5v-uYmsXNAGqk2DoyiTQ>
 
(HttpURLConnection.java:1339) 
at sun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.google.com/url?sa=D&q=www.protocol.http.HttpURLConnection.getInputStream&usg=AFQjCNFqUjkHnP5H1HOyzWN1ONhoAxhpCA>
 
(HttpURLConnection.java:993) 
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier 
$MapOutputCopier.getInputStream(ReduceTask.java:1447) 
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier 
$MapOutputCopier.getMapOutput(ReduceTask.java:1349) 
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier 
$MapOutputCopier.copyOutput(ReduceTask.java:1261) 
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run 
(ReduceTask.java:1195) 
Caused by: java.io.FileNotFoundException: 
http://master:50060/mapOutput?job=job_200912281326_0001&map=attempt_2... <http://www.google.com/url?sa=D&q=http://master:50060/mapOutput%3Fjob%3Djob_200912281326_0001%26map%3Dattempt_200912281326_0001_m_000000_0%26reduce%3D0&usg=AFQjCNHUolZY6jZh4toz_Zsx1tISJS8OUA>
 
at sun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.google.com/url?sa=D&q=www.protocol.http.HttpURLConnection.getInputStream&usg=AFQjCNFqUjkHnP5H1HOyzWN1ONhoAxhpCA>
 
(HttpURLConnection.java:1288) 
... 4 more 


2009-12-28 13:28:36,043 INFO org.apache.hadoop.mapred.ReduceTask: Task 
attempt_200912281326_0001_r_000000_0: Failed fetch #1 from 
attempt_200912281326_0001_m_000000_0 
2009-12-28 13:28:36,043 WARN org.apache.hadoop.mapred.ReduceTask: 
attempt_200912281326_0001_r_000000_0 adding host master to penalty 
box, next contact in 4 seconds 
2009-12-28 13:28:36,044 INFO org.apache.hadoop.mapred.ReduceTask: 
attempt_200912281326_0001_r_000000_0: Got 1 map-outputs from previous 
failures 
2009-12-28 13:28:41,044 INFO org.apache.hadoop.mapred.ReduceTask: 
attempt_200912281326_0001_r_000000_0 Scheduled 1 outputs (0 slow hosts 
and0 dup hosts) 
2009-12-28 13:28:41,054 WARN org.apache.hadoop.mapred.ReduceTask: 
attempt_200912281326_0001_r_000000_0 copy failed: 
attempt_200912281326_0001_m_000000_0 from master 
2009-12-28 13:28:41,058 WARN org.apache.hadoop.mapred.ReduceTask: 
java.io.FileNotFoundException: 
http://master:50060/mapOutput?job=job_200912281326_0001&map=attempt_2... <http://www.google.com/url?sa=D&q=http://master:50060/mapOutput%3Fjob%3Djob_200912281326_0001%26map%3Dattempt_200912281326_0001_m_000000_0%26reduce%3D0&usg=AFQjCNHUolZY6jZh4toz_Zsx1tISJS8OUA>
 
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
Method) 
at sun.reflect.NativeConstructorAccessorImpl.newInstance 
(NativeConstructorAccessorImpl.java:39) 
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance 
(DelegatingConstructorAccessorImpl.java:27) 
at java.lang.reflect.Constructor.newInstance(Constructor.java:513) 
at sun.net.www.protocol.http.HttpURLConnection$6.run <http://www.google.com/url?sa=D&q=www.protocol.http.HttpURLConnection%246.run&usg=AFQjCNF3SFlsqNm7hY5we-D3sXnMDeM1cA>
 
(HttpURLConnection.java:1345) 
at java.security.AccessController.doPrivileged(Native Method) 
at sun.net.www.protocol.http.HttpURLConnection.getChainedException <http://www.google.com/url?sa=D&q=www.protocol.http.HttpURLConnection.getChainedException&usg=AFQjCNFWiaqz2t5v-uYmsXNAGqk2DoyiTQ>
 
(HttpURLConnection.java:1339) 
at sun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.google.com/url?sa=D&q=www.protocol.http.HttpURLConnection.getInputStream&usg=AFQjCNFqUjkHnP5H1HOyzWN1ONhoAxhpCA>
 
(HttpURLConnection.java:993) 
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier 
$MapOutputCopier.getInputStream(ReduceTask.java:1447) 
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier 
$MapOutputCopier.getMapOutput(ReduceTask.java:1349) 
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier 
$MapOutputCopier.copyOutput(ReduceTask.java:1261) 
at org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run 
(ReduceTask.java:1195) 
Caused by: java.io.FileNotFoundException: 
http://master:50060/mapOutput?job=job_200912281326_0001&map=attempt_2... <http://www.google.com/url?sa=D&q=http://master:50060/mapOutput%3Fjob%3Djob_200912281326_0001%26map%3Dattempt_200912281326_0001_m_000000_0%26reduce%3D0&usg=AFQjCNHUolZY6jZh4toz_Zsx1tISJS8OUA>
 
at sun.net.www.protocol.http.HttpURLConnection.getInputStream <http://www.google.com/url?sa=D&q=www.protocol.http.HttpURLConnection.getInputStream&usg=AFQjCNFqUjkHnP5H1HOyzWN1ONhoAxhpCA>
 
(HttpURLConnection.java:1288) 
... 4 more 


and in the hadoop-root-tasktracker shows 


2009-12-28 13:28:34,055 WARN org.apache.hadoop.mapred.TaskTracker: 
getMapOutput(attempt_200912281326_0001_m_000000_0,0) failed : 
org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find 
taskTracker/jobcache/job_200912281326_0001/ 
attempt_200912281326_0001_m_000000_0/output/file.out.index in any of 
the configured local directories 
at org.apache.hadoop.fs.LocalDirAllocator 
$AllocatorPerContext.getLocalPathToRead(LocalDirAllocator.java:389) 
at org.apache.hadoop.fs.LocalDirAllocator.getLocalPathToRead 
(LocalDirAllocator.java:138) 
at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet 
(TaskTracker.java:2886) 
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) 
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) 
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java: 
502) 
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java: 
363) 
at org.mortbay.jetty.security.SecurityHandler.handle 
(SecurityHandler.java:216) 
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java: 
181) 
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java: 
766) 
at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java: 
417) 
at org.mortbay.jetty.handler.ContextHandlerCollection.handle 
(ContextHandlerCollection.java:230) 
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java: 
152) 
at org.mortbay.jetty.Server.handle(Server.java:324) 
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java: 
534) 
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete 
(HttpConnection.java:864) 
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:533) 
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:207) 
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:403) 
at org.mortbay.io.nio.SelectChannelEndPoint.run 
(SelectChannelEndPoint.java:409) 
at org.mortbay.thread.QueuedThreadPool$PoolThread.run 
(QueuedThreadPool.java:522) 


Do you have any idea what could be the problem? 


I'm a newbie in this, can you tell me if hadoop has a forum or 
something similar? 


thanks in advance 
Hervé 

 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message