hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3297) The way in which ReduceTask/TaskTracker gets completion events during shuffle can be improved
Date Mon, 28 Apr 2008 14:21:55 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12592816#action_12592816

Devaraj Das commented on HADOOP-3297:

An interesting observation regarding using the ramfs. I guess i should raise a separate jira
but let me put it here anyway -
I had a job (loadgen from hadoop-test) consisting of 2500 maps and 1 reducer. The ramfs size
was 300MB and io.sort.factor was 100. The cluster had 20 nodes. Each map generated 5 MB of
data. The amount of time it took to complete the job was 45 minutes (with the above changes).
The number of files that missed the ramfs and ended up on disk was ~2000.
I ran the same job (with exactly the same config) with the reducer throttled - if a ramfs
merge is on, it would wait for that to complete before fetching anything new. This basically
results in all files ending up in the ramfs. The job ran in 30 minutes.

So although I didn't notice any significant performance gain for this job with the shuffle
protocol changes as proposed in my last comment but in general it looks like this is going
to be true - for a given job, if we have a faster shuffle, more files get created on the disk,
and depending on the number/size of map outputs for the job, this might adversely affects
the final merge, thereby affecting the overall runtime of the job. 

I will see if the above behavior can be modelled.

> The way in which ReduceTask/TaskTracker gets completion events during shuffle can be
> ---------------------------------------------------------------------------------------------
>                 Key: HADOOP-3297
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3297
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.18.0
> Certain things like poll frequency, number of events fetched in one go, etc. can probably
be improved to improve the shuffle performance. This would affect the task-->tasktracker
and the tasktracker-->jobtracker shuffle related RPCs.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message