hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amar Kamat (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3326) ReduceTask should not sleep for 200 ms while waiting for merge to finish
Date Thu, 22 May 2008 04:45:55 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12598906#action_12598906
] 

Amar Kamat commented on HADOOP-3326:
------------------------------------

Some comments 
1) Make {{copiersDone}} volatile. 
2) I am not sure if changing the ordering of triggering InMem merge and removing it from the
neededOutputs is safe. Sharad, can you plz verify this?
3) Remove extra newlines.
4) Since there is only one thread we can use {{notify()}} instead of {{notifyAll()}}
5) The In-Mem merge trigger condition was 
{code:title=Condition #1 (Before)|borderStyle=solid}
!mergeInProgress && 
(inMemFileSys.getPercentUsed() >= MAX_INMEM_FILESYS_USE || 
               (mergeThreshold > 0 && 
                inMemFileSys.getNumFiles(MAP_OUTPUT_FILTER) >= 
                mergeThreshold))&&
              mergeThrowable == null
{code}
was changed to 
{code:title=Condition #2 (After)|borderStyle=solid}
mergeThreshold<0 && inMemFileSys.getNumFiles(MAP_OUTPUT_FILTER) < mergeThreshold
{code}
 Since condition #1 was the triggering condition and condition #2 is used to wait for the
triggering condition, condition #1 = ~ condition #2. Hence it should be
{code}
 inMemFileSys.getPercentUsed() < MAX_INMEM_FILESYS_USE &&  (mergeThreshold <=
0 || inMemFileSys.getNumFiles(MAP_OUTPUT_FILTER) < mergeThreshold) 
{code}
Or am I missing something?
BTW the approach seems fine.

> ReduceTask should not sleep for 200 ms while waiting for merge to finish
> ------------------------------------------------------------------------
>
>                 Key: HADOOP-3326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3326
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Owen O'Malley
>            Assignee: Sharad Agarwal
>         Attachments: 3326_1.patch
>
>
> Currently the merge code in Reduce task does:
> {code}
>             // Wait for the on-disk merge to complete
>             while (localFSMergeInProgress) {
>               Thread.sleep(200);
>             }
>             
>             //wait for an ongoing merge (if it is in flight) to complete
>             while (mergeInProgress) {
>               Thread.sleep(200);
>             }
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message