hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yogesh Sangvikar <Yogesh_Sangvi...@persistent.co.in>
Subject Help for Tasktracker down with getMapOutput() fails with EofException, followed by IllegalStateException
Date Fri, 21 Sep 2012 09:02:12 GMT
Hi Sir,

I have 4 node hadoop cluster with one of the node acting as master as well as slave node.
The SecondayNameNode is running on one of the slave node.
I am using 8 map (mapred.map.tasks) and 4 reduce (mapred.reduce.tasks) tasks. I am using the
cluster for Nutch 1.4 crawling.
I observed that, one of the slave node (node with secondary Name node) got down with below
Exception in tasktracker log,

2012-09-21 00:37:40,819 WARN  mapred.TaskTracker - getMapOutput(attempt_201209201246_0063_m_000036_0,2)
failed :
org.mortbay.jetty.EofException
       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
       at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:550)
       at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:571)
       at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1005)
       at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:648)
       at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:579)
       at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2940)
....
2012-09-21 00:37:40,820 WARN  mortbay.log - Committed before 410 getMapOutput(attempt_201209201246_0063_m_000036_0,2)
failed :
org.mortbay.jetty.EofException
       at org.mortbay.jetty.HttpGenerator.flush(HttpGenerator.java:787)
       at org.mortbay.jetty.AbstractGenerator$Output.blockForOutput(AbstractGenerator.java:550)
       at org.mortbay.jetty.AbstractGenerator$Output.flush(AbstractGenerator.java:571)
       at org.mortbay.jetty.HttpConnection$Output.flush(HttpConnection.java:1005)
       at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:648)
       at org.mortbay.jetty.AbstractGenerator$Output.write(AbstractGenerator.java:579)
       at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2940)
....
2012-09-21 00:37:40,823 ERROR mortbay.log - /mapOutput
java.lang.IllegalStateException: Committed
       at org.mortbay.jetty.Response.resetBuffer(Response.java:1024)
       at org.mortbay.jetty.Response.sendError(Response.java:240)
       at org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:2963)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
       at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
       at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
       at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:390)


I suspect, above Exceptions has caused the slave node to get down.
I found one reference on JIRA with such issue, but didn't get the root cause and way to resolve
such issues.
https://issues.apache.org/jira/browse/MAPREDUCE-5

Could you help me to understand the root cause of above exceptions and whether this may cause
the Tasktracker to get down?

Your comments will be helpful to understand and give hint to improve the hadoop performance.

Thanks,
Yogesh Sangvikar






DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the property of Persistent
Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed.
If you are not the intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in error, please notify
the sender and delete all copies of this message. Persistent Systems Ltd. does not accept
any liability for virus infected mails.

Mime
View raw message