hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sharad Agarwal (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3517) The last InMemory merge may be missed
Date Mon, 09 Jun 2008 09:03:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603523#action_12603523

Sharad Agarwal commented on HADOOP-3517:

{quote} We probably need to track opened but not yet closed files along with ramManager.close
and then get waitForDataToMerge to return a boolean indicating that the InMemMergeThread should
gracefully exit {quote}
would just calling doInMemMerge after the _while (!exitInMemMerge)_ loop in InMemFSMergeThread

few other observations:
1. The condition in _ShuffleRamManager.reserve_ is not related to condition in _ShuffleRamManager.waitForDataToMerge()_.
So there could be situations (depends on the various thresholds set) when the merge thread
is waiting for thresholds to be met; and at the same time shuffle is stalled waiting for merge
to happen.
2. i think the _RamManger.reserve(int requestedSize, InputStream in)_ interface could be simplied
to _RamManger.reserve(int requestedSize)_. There is no real need to close the connection in
reserve method. The connection could be created after the call to reserve in getMapOutput

> The last InMemory merge may be missed
> -------------------------------------
>                 Key: HADOOP-3517
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3517
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: mapred
>    Affects Versions: 0.18.0
>            Reporter: Devaraj Das
>            Assignee: Arun C Murthy
>            Priority: Blocker
>             Fix For: 0.18.0
> This is post HADOOP-3366. The inmem merge thread has the loop:
> {code}
>         while (!exitInMemMerge) {
>             ramManager.waitForDataToMerge();
>             doInMemMerge();
>           }
> {code}
> The fetchOutputs, at the end of copying everything, does the following:
> {code}
>         exitInMemMerge = true; 
>         ramManager.close();
> {code}
> Now if the merge thread is doing a merge (inside the doInMemMerge method) when the exitInMemMerge
is set to true, the loop will break and the last merge of the files that got shuffled recently
will be skipped. ramManager.close(), that internally does a final notify to the merge thread
also won't have any effect in this case.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message