hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-935) There's little to be gained by putting a host into the penaltybox at reduce time, if its the only host you have
Date Fri, 28 Aug 2009 21:17:32 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12748983#action_12748983
] 

Steve Loughran commented on MAPREDUCE-935:
------------------------------------------

Thread dump of the killed process
{code}
2009-08-28 21:40:39,572 INFO  mapred.Task (ReflectionUtils.java:logThreadInfo(204)) - Process
Thread Dump: Communication exception
14 active threads
Thread 26 (Thread for polling Map Completion Events):
  State: TIMED_WAITING
  Blocked count: 25
  Waited count: 117
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.ipc.Client$Connection.handleConnectionFailure(Client.java:392)
    org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:334)
    org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:197)
    org.apache.hadoop.ipc.Client.getConnection(Client.java:886)
    org.apache.hadoop.ipc.Client.call(Client.java:746)
    org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:223)
    org.apache.hadoop.mapred.$Proxy0.getMapCompletionEvents(Unknown Source)
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.getMapCompletionEvents(ReduceTask.java:2725)
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$GetMapEventsThread.run(ReduceTask.java:2692)
Thread 25 (Thread for merging in memory files):
  State: WAITING
  Blocked count: 12
  Waited count: 13
  Waiting on java.lang.Object@500ab58d
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$ShuffleRamManager.waitForDataToMerge(ReduceTask.java:1092)
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$InMemFSMergeThread.run(ReduceTask.java:2590)
Thread 24 (Thread for merging on-disk files):
  State: WAITING
  Blocked count: 0
  Waited count: 1
  Waiting on java.util.TreeSet@659c2931
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$LocalFSMerger.run(ReduceTask.java:2493)
Thread 23 (MapOutputCopier attempt_20090828213826033_0001_r_000000_0.4):
  State: WAITING
  Blocked count: 267
  Waited count: 266
  Waiting on java.util.ArrayList@15ab7626
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1219)
Thread 22 (MapOutputCopier attempt_20090828213826033_0001_r_000000_0.3):
  State: WAITING
  Blocked count: 273
  Waited count: 270
  Waiting on java.util.ArrayList@15ab7626
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1219)
Thread 21 (MapOutputCopier attempt_20090828213826033_0001_r_000000_0.2):
  State: WAITING
  Blocked count: 274
  Waited count: 275
  Waiting on java.util.ArrayList@15ab7626
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1219)
Thread 20 (MapOutputCopier attempt_20090828213826033_0001_r_000000_0.1):
  State: WAITING
  Blocked count: 257
  Waited count: 257
  Waiting on java.util.ArrayList@15ab7626
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1219)
Thread 19 (MapOutputCopier attempt_20090828213826033_0001_r_000000_0.0):
  State: WAITING
  Blocked count: 77
  Waited count: 72
  Waiting on java.util.ArrayList@15ab7626
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier$MapOutputCopier.run(ReduceTask.java:1219)
Thread 18 (communication thread):
  State: RUNNABLE
  Blocked count: 16
  Waited count: 34
  Stack:
    sun.management.ThreadImpl.getThreadInfo0(Native Method)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:147)
    sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:123)
    org.apache.hadoop.util.ReflectionUtils.printThreadInfo(ReflectionUtils.java:149)
    org.apache.hadoop.util.ReflectionUtils.logThreadInfo(ReflectionUtils.java:203)
    org.apache.hadoop.mapred.Task$TaskReporter.run(Task.java:532)
    java.lang.Thread.run(Thread.java:637)
Thread 15 (Thread for syncLogs):
  State: TIMED_WAITING
  Blocked count: 0
  Waited count: 24
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.mapred.Child$2.run(Child.java:88)
Thread 5 (Signal Dispatcher):
  State: RUNNABLE
  Blocked count: 0
  Waited count: 0
  Stack:
Thread 3 (Finalizer):
  State: WAITING
  Blocked count: 5
  Waited count: 6
  Waiting on java.lang.ref.ReferenceQueue$Lock@21e30857
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Thread 2 (Reference Handler):
  State: WAITING
  Blocked count: 7
  Waited count: 6
  Waiting on java.lang.ref.Reference$Lock@43cda81e
  Stack:
    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:485)
    java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
Thread 1 (main):
  State: TIMED_WAITING
  Blocked count: 16
  Waited count: 26
  Stack:
    java.lang.Thread.sleep(Native Method)
    org.apache.hadoop.mapred.ReduceTask$ReduceCopier.fetchOutputs(ReduceTask.java:2027)
    org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:374)
    org.apache.hadoop.mapred.Child.main(Child.java:162)


{code}

> There's little to be gained by putting a host into the penaltybox at reduce time, if
its the only host you have
> ---------------------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-935
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-935
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: tasktracker
>    Affects Versions: 0.21.0
>            Reporter: Steve Loughran
>
> Exponential backoff may be good for dealing with troublesome hosts, but not if you only
have one host in the entire system. From the log of {{TestNodeRefresh}}, which for some reason
is blocking in the reduce phase, I can see it doesn't take much for the backoff to kick in
so rapidly that the reducer is waiting for longer than the test
> {code}
> 2009-08-28 21:39:16,788 WARN  mapred.ReduceTask (ReduceTask.java:fetchOutputs(2192))
- attempt_20090828213826033_0001_r_000000_0 adding host localhost to penalty box, next contact
in 150 seconds
> {code}
> The result of this backoff process is that the reduce process ends up appearing to hang,
getting killed from above. 
> Note that this isn't the root cause of the problem, but it certainly amplifies things.


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message