hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-331) map outputs should be written to a single output file with an index
Date Wed, 06 Dec 2006 18:15:26 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-331?page=comments#action_12456153 ] 
Owen O'Malley commented on HADOOP-331:

Change the name of the class SorterBase to BufferSorter

Instead of "initialize(JobConf)"  in the file BasicTypeSorterBase.java, it should implement

The BasicTypeSorterBase should be an abstract class since it itself 
doesn't implement the sort algorithm

The sorter should not be concerned about partitions (makes the code 
hard to read). So have one sorter per partition instead of having one sorter 
for all partitions. In the case, where there is one sorter per partition, we 
don't require the callback mechanism (the OutputWriter interface in 
MapTask.java). Also, the ListOfArrays class is not required if this is done.

sort method in the SorterBase interface should return a RawKeyValueIterator
object rather than deriving from it import DataInputBuffer is not used in

SequenceFile.java has some whitespace-only changes

Use readLong/writeLong instead of LongWritables for reading/writing longs

Remove the empty finally block from MapTask.run

Synchronize the collect methods in MapTask.java (move the 'synchronized'
from the method declaration to inside the method as 'synchronized(this)' )

Close the combiner after iterating a sorted partition while spilling to disk.
This is to ensure that in case of streaming, where the combiner is a separate
process, it processes the input key/vals and writes the output key/vals before
the spill file is updated with the partition information.

The occurrences of "sort" strings in ReduceTask.java may remain; they need not
be replaced with "merge" strings (although merge is the more accurate word, but
effectively we are sorting. Also, there may be code somewhere that relies on
status strings being "sort").

In TestMapRed.java, the check whether map output a compressed file or not is
commented out. We need to do the compression check.

> map outputs should be written to a single output file with an index
> -------------------------------------------------------------------
>                 Key: HADOOP-331
>                 URL: http://issues.apache.org/jira/browse/HADOOP-331
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: mapred
>    Affects Versions: 0.3.2
>            Reporter: eric baldeschwieler
>         Assigned To: Devaraj Das
>         Attachments: 331-design.txt, 331-initial3.patch, 331.txt
> The current strategy of writing a file per target map is consuming a lot of unused buffer
space (causing out of memory crashes) and puts a lot of burden on the FS (many opens, inodes
used, etc).  
> I propose that we write a single file containing all output and also write an index file
IDing which byte range in the file goes to each reduce.  This will remove the issue of buffer
waste, address scaling issues with number of open files and generally set us up better for
scaling.  It will also have advantages with very small inputs, since the buffer cache will
reduce the number of seeks needed and the data serving node can open a single file and just
keep it open rather than needing to do directory and open ops on every request.
> The only issue I see is that in cases where the task output is substantiallyu larger
than its input, we may need to spill multiple times.  In this case, we can do a merge after
all spills are complete (or during the final spill).

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message