hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] Updated: (MAPREDUCE-1906) Lower minimum heartbeat interval for tasktracker > Jobtracker
Date Tue, 13 Jul 2010 20:44:52 GMT

     [ https://issues.apache.org/jira/browse/MAPREDUCE-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Scott Carey updated MAPREDUCE-1906:
-----------------------------------

    Status: Patch Available  (was: Open)

MAPREDUCE-1906-0.21-v2.patch

changes ping to a smooth function from a step function and lowers the minimum to 300ms.  
Clusters larger than 300 nodes only see the step-function > smooth function change.  Clusters
between 30 and 300 nodes smoothly increase their ping interval.  Clusters with 30 nodes or
less have 300ms ping intervals when the TT has nothing to do.  This improves scheduling latency
on small clusters significantly.

The cluster wide ping interval is roughly proportional to how fast the cluster can schedule
a job.
|| cluster size || current ping interval (ms) || current ping rate at JT || patched ping interval
(ms) || patched ping rate at JT ||
| 10 | 3000 | 3.33 /sec | 300 | 33.3 /sec |
| 30 | 3000 | 10 /sec | 300 | 100 /sec |
| 100 | 3000 | 33.3 /sec | 1000 | 100 /sec |
| 300 | 3000 | 100 /sec | 3000 | 100 /sec |
| 301 | 4000 | 75 /sec | 3010 | 100 /sec |
| 1000 | 10000 | 100 /sec | 10000 | 100 /sec |
| 1001 | 11000 | 91 /sec | 10010 | 100 /sec |

> Lower minimum heartbeat interval for tasktracker > Jobtracker
> -------------------------------------------------------------
>
>                 Key: MAPREDUCE-1906
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1906
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>    Affects Versions: 0.20.2, 0.20.1
>            Reporter: Scott Carey
>         Attachments: MAPREDUCE-1906-0.21-v2.patch, MAPREDUCE-1906-0.21.patch
>
>
> I get a 0% to 15% performance increase for smaller clusters by making the heartbeat throttle
stop penalizing clusters with less than 300 nodes.
> Between 0.19 and 0.20, the default minimum heartbeat interval increased from 2s to 3s.
  If a JobTracker is throttled at 100 heartbeats / sec for large clusters, why should a cluster
with 10 nodes be throttled to 3.3 heartbeats per second?  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message