hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharad Agarwal (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3517) The last InMemory merge may be missed
Date Tue, 10 Jun 2008 06:49:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603788#action_12603788

Sharad Agarwal commented on HADOOP-3517:

{quote} It probably would work, but as with race-conditions it's hard to prove it works. Either
way, this is hacky at best. {quote}
not sure I understand the possible race condition here. with which thread/threads do you see
the race condition? exitInMemMerge is set to true only after all copier threads are done.

{quote} This definitely should not happen - the idea is that amount of data in-memory and
no. of in-memory file are the only thresholds. 'dataAvailable' and 'close' are the variables
tracking these. {quote}
Thread waiting conditions are based on thresholds (MAX_INMEM_FILESYS_USE, MAX_SINGLE_SHUFFLE_SEGMENT_PERCENT
etc) If you independently tweak these nos, there could be situation for both threads waiting.
consider this case:
mergeThreshold = 0
maxSize = 100
fullSize = size = 65
requestedSize = 36

Wont the shuffle threads and the merge thread, both be waiting for the above case? OR am I
missing anything? 

{quote}No. We need to open to connection to 'figure' requestedSize and we absolutely need
to close the connection if we are waiting for requestedSize to be available...{quote}
Agreed..it seems i missed the requestedSize part. 

> The last InMemory merge may be missed
> -------------------------------------
>                 Key: HADOOP-3517
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3517
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.0
>            Reporter: Devaraj Das
>            Assignee: Arun C Murthy
>            Priority: Blocker
>             Fix For: 0.18.0
> This is post HADOOP-3366. The inmem merge thread has the loop:
> {code}
>         while (!exitInMemMerge) {
>             ramManager.waitForDataToMerge();
>             doInMemMerge();
>           }
> {code}
> The fetchOutputs, at the end of copying everything, does the following:
> {code}
>         exitInMemMerge = true; 
>         ramManager.close();
> {code}
> Now if the merge thread is doing a merge (inside the doInMemMerge method) when the exitInMemMerge
is set to true, the loop will break and the last merge of the files that got shuffled recently
will be skipped. ramManager.close(), that internally does a final notify to the merge thread
also won't have any effect in this case.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message