hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharad Agarwal (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3326) ReduceTask should not sleep for 200 ms while waiting for merge to finish
Date Thu, 05 Jun 2008 15:15:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sharad Agarwal updated HADOOP-3326:
-----------------------------------

    Attachment: 3326_3.patch

Mahadev, Thanks for the review. Have incorporated your comments. Instead of interrupt, using
boolean flags to exit from the thread.
Also, removed the last inMemory merge from fetchOutputs. Now last inMem merge is also handled
by the InMemFSMergeThread.
Tested sort+validate on 50 nodes cluster.

> ReduceTask should not sleep for 200 ms while waiting for merge to finish
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-3326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3326
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Owen O'Malley
>            Assignee: Sharad Agarwal
>             Fix For: 0.18.0
>
>         Attachments: 3326_1.patch, 3326_2.patch, 3326_3.patch
>
>
> Currently the merge code in Reduce task does:
> {code}
>             // Wait for the on-disk merge to complete
>             while (localFSMergeInProgress) {
>               Thread.sleep(200);
>             }
>             
>             //wait for an ongoing merge (if it is in flight) to complete
>             while (mergeInProgress) {
>               Thread.sleep(200);
>             }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message