hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Venkat Seeth <sv...@yahoo.com>
Subject Reduce hangs at times
Date Sat, 24 Feb 2007 17:32:09 GMT
Hi there,

Howdy. I observe at times that few of the reduce tasks
hangs during copy phase and does not result in
failures also. Hence these tasks never complete nor
rerun for timeouts.

reduce > copy (1510 of 1540 at 1.57 MB/s) >

At the same time, I see that Jetty is out of threads
in its thread pool. Dont know if these 2 are related.

I also see the following exception for many of the MR
operations.

24 Feb 2007 02:52:14,438  WARN - No thread for
Socket[addr=/10.163.63.137,port=56019,localport=50060]
- at
org.mortbay.util.ThreadPool.run(ThreadPool.java:373)
24 Feb 2007 02:52:24,440  WARN - No thread for
Socket[addr=/10.163.63.137,port=56023,localport=50060]
- at
org.mortbay.util.ThreadPool.run(ThreadPool.java:373)
24 Feb 2007 02:53:01,582  WARN -
getMapOutput(task_0005_m_000432_0,30) failed :
java.net.SocketException: Connection reset
        at
java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:96)
        at
java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at
org.mortbay.http.ChunkingOutputStream.bypassWrite(ChunkingOutputStream.java:151)
        at
org.mortbay.http.BufferedOutputStream.write(BufferedOutputStream.java:139)
        at
org.mortbay.http.HttpOutputStream.write(HttpOutputStream.java:423)
        at
org.mortbay.jetty.servlet.ServletOut.write(ServletOut.java:54)
        at
org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1574)
        at
javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
        at
javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
        at
org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:427)
        at
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java:475)
        at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:567)
        at
org.mortbay.http.HttpContext.handle(HttpContext.java:1565)
        at
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java:635)
        at
org.mortbay.http.HttpContext.handle(HttpContext.java:1517)
        at
org.mortbay.http.HttpServer.service(HttpServer.java:954)
        at
org.mortbay.http.HttpConnection.service(HttpConnection.java:814)
        at
org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:981)
        at
org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
        at
org.mortbay.http.SocketListener.handleConnection(SocketListener.java:244)
        at
org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
        at
org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
 - at
org.apache.hadoop.mapred.TaskTracker.doGet(TaskTracker.java:1600)
24 Feb 2007 02:53:01,583  WARN -
getMapOutput(task_0005_m_001465_0,55) failed :

Has anyone experienced this? Any thoughts are greatly
appreciated.

Thanks,
Venkat



 
____________________________________________________________________________________
Don't pick lemons.
See all the new 2007 cars at Yahoo! Autos.
http://autos.yahoo.com/new_cars.html 

Mime
View raw message