hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arun C Murthy (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3517) The last InMemory merge may be missed
Date Thu, 12 Jun 2008 04:38:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12604411#action_12604411
] 

Arun C Murthy commented on HADOOP-3517:
---------------------------------------

bq.  -1 findbugs. The patch appears to introduce 1 new Findbugs warnings.

Findbugs seems confused... http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2642/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
says:

{quote}
IS 	Inconsistent synchronization of org.apache.hadoop.mapred.ReduceTask$ReduceCopier$ShuffleRamManager.numPendingRequests;
locked 66% of time
	

Bug type IS2_INCONSISTENT_SYNC (click for details)
In class org.apache.hadoop.mapred.ReduceTask$ReduceCopier$ShuffleRamManager
Field org.apache.hadoop.mapred.ReduceTask$ReduceCopier$ShuffleRamManager.numPendingRequests
Synchronized 66% of the time
Unsynchronized access at ReduceTask.java:[line 793]
Unsynchronized access at ReduceTask.java:[line 793]
Synchronized access at ReduceTask.java:[line 723]
Synchronized access at ReduceTask.java:[line 753]
Synchronized access at ReduceTask.java:[line 753]
Synchronized access at ReduceTask.java:[line 762]
Synchronized access at ReduceTask.java:[line 762]
{quote}

which isn't true! numPendingRequests is always guarded by 'dataAvailable'.

> The last InMemory merge may be missed
> -------------------------------------
>
>                 Key: HADOOP-3517
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3517
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.0
>            Reporter: Devaraj Das
>            Assignee: Arun C Murthy
>            Priority: Blocker
>             Fix For: 0.18.0
>
>         Attachments: HADOOP-3517_1_20080610.patch, HADOOP-3517_2_20080611.patch, HADOOP-3517_2_20080611.patch
>
>
> This is post HADOOP-3366. The inmem merge thread has the loop:
> {code}
>         while (!exitInMemMerge) {
>             ramManager.waitForDataToMerge();
>             doInMemMerge();
>           }
> {code}
> The fetchOutputs, at the end of copying everything, does the following:
> {code}
>         exitInMemMerge = true; 
>         ramManager.close();
> {code}
> Now if the merge thread is doing a merge (inside the doInMemMerge method) when the exitInMemMerge
is set to true, the loop will break and the last merge of the files that got shuffled recently
will be skipped. ramManager.close(), that internally does a final notify to the merge thread
also won't have any effect in this case.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message