hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharad Agarwal (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-3326) ReduceTask should not sleep for 200 ms while waiting for merge to finish
Date Wed, 21 May 2008 12:03:55 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sharad Agarwal updated HADOOP-3326:
-----------------------------------

    Attachment: 3326_1.patch

Attaching a patch. It does following:-
1. LocalFSMerger and InMemFSMergeThread threads are spawned only once in fetchOutputs() method.
The thread gets notified when something is ready for merge. The merge happens when thresholds
are met.

2. Instead of sleeping (in loop) to wait for finishing the merger threads, now it does join.


3. The in memory merge code was duplicate at 2 places, now the common code has been moved
to doInMemMerge() method.

> ReduceTask should not sleep for 200 ms while waiting for merge to finish
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-3326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3326
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Owen O'Malley
>            Assignee: Sharad Agarwal
>         Attachments: 3326_1.patch
>
>
> Currently the merge code in Reduce task does:
> {code}
>             // Wait for the on-disk merge to complete
>             while (localFSMergeInProgress) {
>               Thread.sleep(200);
>             }
>             
>             //wait for an ongoing merge (if it is in flight) to complete
>             while (mergeInProgress) {
>               Thread.sleep(200);
>             }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message