hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Devaraj Das (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-1027) Fix the RAM FileSystem/Merge problems (reported in HADOOP-1014)
Date Tue, 20 Feb 2007 13:01:51 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-1027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Devaraj Das updated HADOOP-1027:

    Attachment: 1027.patch

The attached patch does the following:
1) Modifies hadoop-default.xml; sets the value of fs.inmemory.size.mb to 75

2) SequenceFile.Sorter.MergeQueue.merge is modified to cleanup empty segments right at the
beginning where the segments are obtained from the sorted map datastructure. Segments are
extracted from the sorted map until we have collected the desired number of segments to merge
or have no more segments to look at. This optimizes cases where we have lots of empty segments
to merge.

3) InMemoryFileSystem class has been modified to have synchronization blocks wherever pathToFileAttribs
map is touched. Some of them may be unnecessary but don't harm. It also has what Owen pointed
out as a comment on HADOOP-1014. renameRaw has also been modified to throw an exception if
the dst already exists.

4) The changes in ReduceTaskRunner are:
  4.1) MERGE_THRESHOLD, a number signifying the limit on the max number of files we will accumulate
before initiating inmem merge, has been introduced. This is set to 500.
  4.2) The check for whether to initiate inmem merge has been modified to take into account
  4.3) The on-disk spill file is now created prior to invoking merge (to take care of the
case where we may not have any files left in the ram fs to cloneFileAttributes from; this
will happen if all map outputs are empty)
  4.4) InMemFileSys.close is called at just one place now

> Fix the RAM FileSystem/Merge problems (reported in HADOOP-1014)
> ---------------------------------------------------------------
>                 Key: HADOOP-1027
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1027
>             Project: Hadoop
>          Issue Type: Bug
>          Components: mapred
>            Reporter: Devaraj Das
>         Assigned To: Devaraj Das
>         Attachments: 1027.patch
> 1) Merge algorithm implementation does not delete empty segments (sequence files with
no key/val data) in cases where single level merges don't happen on those segments (due to
the check "numberOfSegmentsRemaining <= factor" returning true). This affected the in-mem
merge in a subtle way :-
> For the in-mem merge, the merge-spill-file is given the same name as the name of the
0th entry file in the ramfs. If this file was an empty file, then it would not get deleted
from the ramfs, and if the subsequent merge on ramfs chose the same name for the merge-spill-file,
it would overwrite the previously created spill. This led to the inconsistent output sizes.
> 2) The InMemoryFileSystem has a "close" method which is not protected (only method where
pathToFileAttribs map is modified without first locking the InMemoryFileSystem instance) and
that quite likely leads to ConcurrentModificationException if some thread calls InMemoryFileSystem.close
(due to some exception) and some other thread is in the process of doing InMemoryFileSystem.getFiles().
However, this problem will not affect the correctness of the merge process (anyway the task
is going to fail) and the more important thing is that some other exception happened (like
insufficient disk space and so map outputs could not be written) which may not be related
to the merge process at all.
> 3) The number of outputs that is merged at once in RAM should be limited. This is to
prevent OutOfMemory errors. Consider a case where there are 10s of thousands of maps and all
maps generate empty outputs. Given the default size of the RAM FS as 75 MB, we can possibly
accomodate lots of map outputs in RAM without doing any merge but it also results in the various
other data structures exploding in size. We have to do a trade off here especially because
the inmem-merging is done in the TaskTracker process which already is under a good amount
of memory pressure.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message