hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@openindex.io>
Subject Re: Error reading task output and (log)FileNotFoundExceptions
Date Mon, 23 Jan 2012 14:26:48 GMT
I took a look at it again and noticed that the mappers fail when data is not 
local. Also, the number of failures of this type is always equal to the number 
of rack-local maps.

So, it seems that when data is not local my mapper fails. It makes sense that 
we only notice it when we have concurrent jobs running.

Any hints to share?

On Tuesday 03 January 2012 22:55:08 Markus Jelsma wrote:
> Hi,
> 
> On our 0.20.205.0 test cluster we sometimes see tasks failing for no clear
> reason. The task tracker logs show us:
> 
> 2012-01-03 21:16:27,256 WARN org.apache.hadoop.mapred.TaskLog: Failed to
> retrieve stdout log for task: attempt_201201031651_0008_m_000233_0
> java.io.FileNotFoundException:
> /opt/hadoop/hadoop-0.20.205.0/libexec/../logs/userlogs/job_201201031651_000
> 8/attempt_201201031651_0008_m_000233_0/log.index (No such file or
> directory)
>         at java.io.FileInputStream.open(Native Method)
>         at java.io.FileInputStream.<init>(FileInputStream.java:120)
>         at
> org.apache.hadoop.io.SecureIOUtils.openForRead(SecureIOUtils.java:102)
>         at
> org.apache.hadoop.mapred.TaskLog.getAllLogsFileDetails(TaskLog.java:187)
>         at org.apache.hadoop.mapred.TaskLog$Reader.<init>(TaskLog.java:422)
>         at
> org.apache.hadoop.mapred.TaskLogServlet.printTaskLog(TaskLogServlet.java:81
> ) at
> org.apache.hadoop.mapred.TaskLogServlet.doGet(TaskLogServlet.java:296)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>         at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
>         at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle
> r.java:1221) at
> org.apache.hadoop.http.HttpServer$QuotingInputFilter.doFilter(HttpServer.ja
> va:835) at
> org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandle
> r.java:1212) at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
>         at
> org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
>         at
> org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
>         at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
>         at
> org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
>         at
> org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCol
> lection.java:230) at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>         at org.mortbay.jetty.Server.handle(Server.java:326)
>         at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
>         at
> org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnecti
> on.java:928)
> 
> However, if we inspect the log more closely we actually see it happening
> several times but only one seems to be thrown simultaneously with the task
> failing. We see no errors in the datanode's log.
> 
> I've been looking through the configuration descriptions of 0.20.205.0 but
> didn't find a setting that could be fix this or is responsible for this.
> 
> Any hints? Upgrade to 1.0? Patches?
> Thanks

-- 
Markus Jelsma - CTO - Openindex

Mime
View raw message