hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Baldeschwieler <eri...@yahoo-inc.com>
Subject Re: [jira] Commented: (HADOOP-331) map outputs should be written to a single output file with an index
Date Fri, 30 Jun 2006 21:13:33 GMT

Merging on the mapper would certainly make sense in that scenario.

On Jun 29, 2006, at 10:32 AM, paul sutter (JIRA) wrote:

>     [ http://issues.apache.org/jira/browse/HADOOP-331? 
> page=comments#action_12418501 ]
> paul sutter commented on HADOOP-331:
> ------------------------------------
> one more suggestion (plea): could the map-output-spill-extraction  
> take place on the mapper side (perhaps in the http server), such  
> that the reducer be able to retreive fully-formed map outputs as  
> currently?
> incidently, we have individual map inputs that are 5-10GB in size,  
> which are compressed so they cant be split, which would yield  
> potentially 50-100 spills per mapper. it would be nice if the  
> reducer (or whatever process is fetching the files) didnt have to  
> do that reassembly, but rather that it happened on the mapper side.
>> map outputs should be written to a single output file with an index
>> -------------------------------------------------------------------
>>          Key: HADOOP-331
>>          URL: http://issues.apache.org/jira/browse/HADOOP-331
>>      Project: Hadoop
>>         Type: Improvement
>>   Components: mapred
>>     Versions: 0.3.2
>>     Reporter: eric baldeschwieler
>>     Assignee: Yoram Arnon
>>      Fix For: 0.5.0
>> The current strategy of writing a file per target map is consuming  
>> a lot of unused buffer space (causing out of memory crashes) and  
>> puts a lot of burden on the FS (many opens, inodes used, etc).
>> I propose that we write a single file containing all output and  
>> also write an index file IDing which byte range in the file goes  
>> to each reduce.  This will remove the issue of buffer waste,  
>> address scaling issues with number of open files and generally set  
>> us up better for scaling.  It will also have advantages with very  
>> small inputs, since the buffer cache will reduce the number of  
>> seeks needed and the data serving node can open a single file and  
>> just keep it open rather than needing to do directory and open ops  
>> on every request.
>> The only issue I see is that in cases where the task output is  
>> substantiallyu larger than its input, we may need to spill  
>> multiple times.  In this case, we can do a merge after all spills  
>> are complete (or during the final spill).
> -- 
> This message is automatically generated by JIRA.
> -
> If you think it was sent incorrectly contact one of the  
> administrators:
>    http://issues.apache.org/jira/secure/Administrators.jspa
> -
> For more information on JIRA, see:
>    http://www.atlassian.com/software/jira

View raw message