hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "seok keun oh" <ohs...@gmail.com>
Subject map-reduce struck...what's wrong?
Date Thu, 15 Feb 2007 01:45:48 GMT
I just run simple map reduce task. (identity mapper, identity reducer)
Job progress reach map 100% reduce 89~92%...
Job hanging...
what's wrong.

version : hadoop 0.8.0
os : FreeBSD 6.2 64bit
data size : 886M
data type : org.apache.nutch.protocol.Content
box size : 9 (1 jobtracker box, 7 tasktracker box, all datanode, namenode
running another box)
mapred.map.tasks : 140
mapred.reduce.tasks : 14


thread dump :
2007-02-15 10:14:11,904 INFO  mapred.TaskTracker - Process Thread Dump: lost
task
20 active threads
Thread 320 (IPC Client connection to xc1/10.8.50.31:9001):
  State: RUNNABLE
  Blocked count: 1
  Waited count: 0
  Stack:
    org.apache.hadoop.ipc.Client$Call.callComplete(Client.java:89)
    org.apache.hadoop.ipc.Client$Connection.run(Client.java:265)
Thread 258 (Thread-223):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 257 (process reaper):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 255 (Thread-221):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.access$100(TaskRunner.java:36)
    org.apache.hadoop.mapred.TaskRunner$1.run(TaskRunner.java:304)
Thread 254 (process reaper):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.UNIXProcess.waitForProcessExit(Native Method)
    java.lang.UNIXProcess.access$900(UNIXProcess.java:20)
    java.lang.UNIXProcess$1$1.run(UNIXProcess.java:132)
Thread 239 (Thread-206):
  State: RUNNABLE
  Blocked count: 36
  Waited count: 147
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 230 (Thread-198):
  State: RUNNABLE
  Blocked count: 30
  Waited count: 141
  Stack:
    java.io.FileInputStream.readBytes(Native Method)
    java.io.FileInputStream.read(FileInputStream.java:194)
    java.io.BufferedInputStream.read1(BufferedInputStream.java:254)
    java.io.BufferedInputStream.read(BufferedInputStream.java:313)
    sun.nio.cs.StreamDecoder$CharsetSD.readBytes(StreamDecoder.java:411)
    sun.nio.cs.StreamDecoder$CharsetSD.implRead(StreamDecoder.java:453)
    sun.nio.cs.StreamDecoder.read(StreamDecoder.java:183)
    java.io.InputStreamReader.read(InputStreamReader.java:167)
    java.io.BufferedReader.fill(BufferedReader.java:136)
    java.io.BufferedReader.readLine(BufferedReader.java:299)
    java.io.BufferedReader.readLine(BufferedReader.java:362)
    org.apache.hadoop.mapred.TaskRunner.logStream(TaskRunner.java:340)
    org.apache.hadoop.mapred.TaskRunner.runChild(TaskRunner.java:308)
    org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:198)
Thread 228 (SocketListener0-6):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 63
  Stack:
    java.net.SocketInputStream.socketRead0(Native Method)
    java.net.SocketInputStream.read(SocketInputStream.java:129)
    org.mortbay.util.LineInput.fill(LineInput.java:469)
    org.mortbay.util.LineInput.fillLine(LineInput.java:547)
    org.mortbay.util.LineInput.readLineBuffer(LineInput.java:293)
    org.mortbay.util.LineInput.readLineBuffer(LineInput.java:277)
    org.mortbay.http.HttpRequest.readHeader(HttpRequest.java:238)
    org.mortbay.http.HttpConnection.readRequest(HttpConnection.java:861)
    org.mortbay.http.HttpConnection.handleNext(HttpConnection.java:907)
    org.mortbay.http.HttpConnection.handle(HttpConnection.java:831)
    org.mortbay.http.SocketListener.handleConnection(SocketListener.java
:244)
    org.mortbay.util.ThreadedServer.handle(ThreadedServer.java:357)
    org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java:534)
Thread 20 (org.apache.hadoop.dfs.DFSClient$LeaseChecker@7a0d85cc):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.dfs.DFSClient$LeaseChecker.run(DFSClient.java:462)
    java.lang.Thread.run(Thread.java:595)
Thread 16 (org.apache.hadoop.io.ObjectWritable Connection Culler):
  State: TIMED_WAITING
  Blocked count: 2
  Waited count: 0
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.ipc.Client$ConnectionCuller.run(Client.java:388)
Thread 15 (IPC Server handler 1 on 50050):
  State: TIMED_WAITING
  Blocked count: 2
  Waited count: 732
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:490)
Thread 14 (IPC Server handler 0 on 50050):
  State: TIMED_WAITING
  Blocked count: 3
  Waited count: 732
  Stack:
    java.lang.Object.wait(Native Method)
    org.apache.hadoop.ipc.Server$Handler.run(Server.java:490)
Thread 13 (IPC Server listener on 50050):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    sun.nio.ch.PollArrayWrapper.poll0(Native Method)
    sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
    sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
    sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
    sun.nio.ch.SelectorImpl.select(SelectorImpl.java:84)
    org.apache.hadoop.ipc.Server$Listener.run(Server.java:224)
Thread 11 (Acceptor ServerSocket[addr=
0.0.0.0/0.0.0.0,port=0,localport=50060]):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
    java.net.PlainSocketImpl.socketAccept(Native Method)
    java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
    java.net.ServerSocket.implAccept(ServerSocket.java:450)
    java.net.ServerSocket.accept(ServerSocket.java:421)
    org.mortbay.util.ThreadedServer.acceptSocket(ThreadedServer.java:432)
    org.mortbay.util.ThreadedServer$Acceptor.run(ThreadedServer.java:631)
Thread 10 (SessionScavenger):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 0
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.mortbay.jetty.servlet.AbstractSessionManager$SessionScavenger.run(
AbstractSessionManager.java:587)
Thread 9 (taskCleanup):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.ArrayList@3adec8b3
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:474)
    org.apache.hadoop.mapred.TaskTracker$BlockingQueue.take(TaskTracker.java
:783)
    org.apache.hadoop.mapred.TaskTracker$1.run(TaskTracker.java:143)
    java.lang.Thread.run(Thread.java:595)
Thread 4 (Signal Dispatcher):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
Thread 3 (Finalizer):
  State: WAITING
  Blocked count: 0
  Waited count: 24
  Waiting on java.lang.ref.ReferenceQueue$Lock@784a7df6
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
  State: WAITING
  Blocked count: 3
  Waited count: 25
  Waiting on java.lang.ref.Reference$Lock@6293df2c
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:474)
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
  State: RUNNABLE
  Blocked count: 19
  Waited count: 1499
  Stack:
    sun.management.ThreadImpl.getThreadInfo0(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:144)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:120)
    org.apache.hadoop.util.ReflectionUtils.printThreadInfo(
ReflectionUtils.java:93)
    org.apache.hadoop.util.ReflectionUtils.logThreadInfo(
ReflectionUtils.java:141)
    org.apache.hadoop.mapred.TaskTracker.markUnresponsiveTasks(
TaskTracker.java:553)
    org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:449)
    org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:720)
    org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:1374)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message