hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Resolved: (HADOOP-165) long response times from task trackers under load
Date Thu, 24 May 2007 06:12:16 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Owen O'Malley resolved HADOOP-165.
----------------------------------

    Resolution: Duplicate
      Assignee:     (was: Owen O'Malley)

This has been fixed primarily by moving the pulling part of the shuffle from the task tracker
to the reduce's jvm. Previously, we saw task trackers burning 100% cpu and now it is much
better.

> long response times from task trackers under load
> -------------------------------------------------
>
>                 Key: HADOOP-165
>                 URL: https://issues.apache.org/jira/browse/HADOOP-165
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.2.0
>         Environment: 190 redhat linux boxes, each with 2 hyperthreaded cpus, 2gb memory
>            Reporter: Owen O'Malley
>
> we are seeing very slow response times from the task tracker. I put in some instrumentation
to measure how long each call took for the RPC.Sever code to run the method (so it does not
include serialization/deserialization time). The top of the list (in ms) looks like:
> 7581 progress node1192
> 7022 ping node1192
> 5393 ping node1162
> 4854 progress node1162
> 4749 progress node1194
> 4709 ping node1194
> 3813 ping node1100
> 3486 ping node1190
> 3266 progress node1190
> 3187 progress node1265
> 3078 ping node1203
> 2972 progress node1203
> 2947 progress node1240
> 2889 progress node1100
> 2875 ping node1116
> 2843 ping node1189
> 2772 ping node1183
> 2737 ping node1110
> 2727 progress node1183
> 2710 ping node1123
> 2563 ping node1304
> 2527 progress node1144
> 2479 ping node1137
> 2476 progress node1304
> 2430 ping node1240
> 2416 ping node1144
> 2377 progress node1176
> 2339 ping node1109
> 2321 progress node1114
> 2311 ping node1157
> 2185 ping node1265
> 2185 ping node1109
> 2172 ping node1114
> 2145 progress node1109
> 2127 ping node1176
> 2083 progress node1189
> 2076 ping node1229
> 2073 progress node1188
> 2072 progress node1123
> 2048 ping node1161
> 2003 progress node1110
> 1989 ping node1180
> 1963 ping node1114

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message