hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johan Oskarson (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-707) Final map task gets stuck
Date Fri, 17 Nov 2006 17:23:40 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-707?page=comments#action_12450802 ] 
            
Johan Oskarson commented on HADOOP-707:
---------------------------------------

Same problem, different job.
Full thread dump.

2006-11-17 15:08:14,707 INFO org.apache.hadoop.mapred.TaskTracker: task_0133_m_000948_0: Task
failed to report status for 608 seconds. Killing.
2006-11-17 15:08:14,722 INFO org.apache.hadoop.mapred.TaskTracker: Process Thread Dump: lost
task
65 active threads
Thread 46827 (IPC Client connection to master1.herd1/10.0.5.10:9001):
  State: WAITING
  Blocked count: 1
  Waited count: 2
  Waiting on org.apache.hadoop.ipc.Client$Connection@74e65f
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:474)
    org.apache.hadoop.ipc.Client$Connection.waitForWork(Client.java:195)
    org.apache.hadoop.ipc.Client$Connection.run(Client.java:234)
Thread 46826 (btpool0-893):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46825 (btpool0-892):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46822 (btpool0-891):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 1
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46811 (btpool0-890):
  State: RUNNABLE
  Blocked count: 3
  Waited count: 1
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46810 (btpool0-889):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46803 (btpool0-888):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 8
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46408 (btpool0-887):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 41
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46402 (btpool0-886):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 26
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46396 (btpool0-885):
  State: RUNNABLE
  Blocked count: 4
  Waited count: 48
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(LocalFileSystem.java:113)
    org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:91)
    org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:189)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    java.io.DataInputStream.read(DataInputStream.java:80)
    org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1411)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
    org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
    org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
    org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
    org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
    org.mortbay.jetty.Server.handle(Server.java:269)
    org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
    org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:687)
Thread 46345 (btpool0-883):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 26
  Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:469)
Thread 46333 (btpool0-881):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 42
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46305 (btpool0-880):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 43
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46304 (btpool0-879):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 37
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46295 (btpool0-878):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 36
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46253 (btpool0-877):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 48
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46252 (btpool0-876):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 46
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46251 (btpool0-875):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 51
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
   org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 46242 (btpool0-874):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 43
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 45039 (Thread-28307):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45038 (process reaper):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 45031 (Thread-28302):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45030 (process reaper):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 45026 (Thread-28299):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 45025 (process reaper):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44998 (Thread-28283):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 44997 (process reaper):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44987 (Thread-28277):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 44986 (process reaper):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 44905 (btpool0-872):
  State: TIMED_WAITING
  Blocked count: 2
  Waited count: 87
  Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:469)
Thread 44903 (btpool0-870):
  State: TIMED_WAITING
  Blocked count: 1
  Waited count: 103
  Stack:
    java.lang.Object.wait(Native Method)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:469)
Thread 44891 (btpool0-865):
  State: RUNNABLE
  Blocked count: 5
  Waited count: 55
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44888 (btpool0-863):
  State: RUNNABLE
  Blocked count: 7
  Waited count: 135
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44887 (btpool0-862):
  State: RUNNABLE
  Blocked count: 4
  Waited count: 131
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44882 (btpool0-857):
  State: RUNNABLE
  Blocked count: 9
  Waited count: 138
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44881 (btpool0-856):
  State: RUNNABLE
  Blocked count: 3
  Waited count: 127
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44841 (btpool0-851):
  State: RUNNABLE
  Blocked count: 6
  Waited count: 127
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 44752 (btpool0-849):
  State: RUNNABLE
  Blocked count: 7
  Waited count: 134
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 43631 (Thread-27401):
  State: RUNNABLE
  Blocked count: 114
  Waited count: 1730
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43610 (Thread-27384):
  State: RUNNABLE
  Blocked count: 116
  Waited count: 1687
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43591 (Thread-27369):
  State: RUNNABLE
  Blocked count: 79
  Waited count: 1690
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43583 (Thread-27361):
  State: RUNNABLE
  Blocked count: 109
  Waited count: 1690
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43575 (Thread-27353):
  State: RUNNABLE
  Blocked count: 128
  Waited count: 1726
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 43052 (btpool0-844):
  State: RUNNABLE
  Blocked count: 6
  Waited count: 217
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 43049 (btpool0-842):
  State: RUNNABLE
  Blocked count: 4
  Waited count: 213
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 43042 (btpool0-835):
  State: RUNNABLE
  Blocked count: 12
  Waited count: 229
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(LocalFileSystem.java:113)
    org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:189)
    java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    java.io.DataInputStream.readFully(DataInputStream.java:176)
    java.io.DataInputStream.readFully(DataInputStream.java:152)
    org.apache.hadoop.fs.FSDataInputStream$Checker.<init>(FSDataInputStream.java:60)
    org.apache.hadoop.fs.FSDataInputStream.<init>(FSDataInputStream.java:279)
    org.apache.hadoop.fs.FileSystem.open(FileSystem.java:156)
    org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1409)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
    org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
    org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
    org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
Thread 40895 (btpool0-803):
  State: RUNNABLE
  Blocked count: 3
  Waited count: 361
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 40380 (btpool0-790):
  State: RUNNABLE
  Blocked count: 7
  Waited count: 391
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    org.apache.hadoop.fs.LocalFileSystem$LocalFSFileInputStream.read(LocalFileSystem.java:113)
    org.apache.hadoop.fs.FSDataInputStream$Checker.read(FSDataInputStream.java:91)
    org.apache.hadoop.fs.FSDataInputStream$PositionCache.read(FSDataInputStream.java:189)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    java.io.DataInputStream.read(DataInputStream.java:80)
    org.apache.hadoop.mapred.TaskTracker$MapOutputServlet.doGet(TaskTracker.java:1411)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
    javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
    org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:445)
    org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:356)
    org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:226)
    org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:627)
    org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)
    org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:141)
    org.mortbay.jetty.Server.handle(Server.java:269)
    org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:430)
    org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:687)
Thread 39121 (btpool0-784):
  State: RUNNABLE
  Blocked count: 16
  Waited count: 439
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 39112 (btpool0-775):
  State: RUNNABLE
  Blocked count: 13
  Waited count: 396
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.io.ByteArrayBuffer.readFrom(ByteArrayBuffer.java:168)
    org.mortbay.io.bio.StreamEndPoint.fill(StreamEndPoint.java:98)
    org.mortbay.jetty.bio.SocketConnector$Connection.fill(SocketConnector.java:181)
    org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:263)
    org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:193)
    org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:339)
    org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:208)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 28 (org.apache.hadoop.dfs.DFSClient$LeaseChecker@1a7ddcf):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:462)
    java.lang.Thread.run(Thread.java:595)
Thread 19 (org.apache.hadoop.io.ObjectWritable Connection Culler):
  State: TIMED_WAITING
  Blocked count: 154
  Waited count: 0
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.ipc.Client$ConnectionCuller.run(Client.java:388)
Thread 18 (IPC Server handler 4 on 50050):
  State: TIMED_WAITING
  Blocked count: 1769
  Waited count: 51161
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 17 (IPC Server handler 3 on 50050):
  State: TIMED_WAITING
  Blocked count: 1673
  Waited count: 51202
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 16 (IPC Server handler 2 on 50050):
  State: TIMED_WAITING
  Blocked count: 1685
  Waited count: 51188
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 15 (IPC Server handler 1 on 50050):
  State: TIMED_WAITING
  Blocked count: 1694
  Waited count: 51183
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 14 (IPC Server handler 0 on 50050):
  State: TIMED_WAITING
  Blocked count: 1913
  Waited count: 51137
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:486)
Thread 13 (IPC Server listener on 50050):
  State: RUNNABLE
  Blocked count: 634
  Waited count: 0
  Stack:
    sun.nio.ch.PollArrayWrapper.poll0(Native Method)
    sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
    sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:224)
Thread 12 (btpool0-1 - Acceptor0 SocketConnector @ 0.0.0.0:50060):
  State: RUNNABLE
  Blocked count: 85
  Waited count: 1517
  Stack:
    java.net.PlainSocketImpl.socketAccept(Native Method)
    java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    java.net.ServerSocket.implAccept(ServerSocket.java:450)
    java.net.ServerSocket.accept(ServerSocket.java:421)
    org.mortbay.jetty.bio.SocketConnector.accept(SocketConnector.java:97)
    org.mortbay.jetty.AbstractConnector$Acceptor.run(AbstractConnector.java:630)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 11 (btpool0-0 - Invalidator - /):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 1
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(AbstractSessionManager.java:933)
    org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:475)
Thread 9 (taskCleanup):
  State: WAITING
  Blocked count: 457
  Waited count: 118
  Waiting on java.util.ArrayList@15192a
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:474)
    org.apache.hadoop.mapred.TaskTracker$BlockingQueue.take(TaskTracker.java:783)
    org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:143)
    java.lang.Thread.run(Thread.java:595)
Thread 4 (Signal Dispatcher):
  State: RUNNABLE
  Blocked count: 0
 Waited count: 0
  Stack:
Thread 3 (Finalizer):
  State: WAITING
  Blocked count: 99047
  Waited count: 39648
  Waiting on java.lang.ref.ReferenceQueue$Lock@ac6214
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
  State: WAITING
  Blocked count: 102986
  Waited count: 3131
  Waiting on java.lang.ref.Reference$Lock@15b40a1
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:474)
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
  State: RUNNABLE
  Blocked count: 1538
  Waited count: 107465
  Stack:
    sun.management.ThreadImpl.getThreadInfo0(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:144)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120)
    org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:93)
    org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:141)
    org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(TaskTracker.java:553)
    org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:449)
    org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:720)
    org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1374)


> Final map task gets stuck
> -------------------------
>
>                 Key: HADOOP-707
>                 URL: http://issues.apache.org/jira/browse/HADOOP-707
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.8.0
>         Environment: using latest trunk
>            Reporter: Johan Oskarson
>            Priority: Critical
>
> I've seen numerous jobs lately where the final map task gets stuck, never finishing.
> The jobtracker doesn't reassign the task. A restart of the tasktracker solves the issue
and the job can finish.
> In the web interface it turns up as:
> task_0028_m_000534_0 node17.herd1 RUNNING 0.00%    10-Nov-2006 12:21:12 10-Nov-2006 12:22:19
(1mins, 6sec)
> Task failed to report status for 604 seconds. Killing.
> Only exception I find in that tasktracker log is this (a few times):
> java.nio.channels.ClosedChannelException
>         at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
>         at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
>         at org.apache.hadoop.ipc.SocketChannelOutputStream.flushBuffer(SocketChannelOutputStream.java:108)
>         at org.apache.hadoop.ipc.SocketChannelOutputStream.write(SocketChannelOutputStream.java:89)
>         at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
>         at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
>         at java.io.DataOutputStream.flush(DataOutputStream.java:106)
>         at org.apache.hadoop.ipc.Server$Handler.run(Server.java:532)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message