hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Billy Pearson" <sa...@pearsonwholesale.com>
Subject Re: Is intermediate data produced by mappers always flushed to disk ?
Date Tue, 19 May 2009 19:55:53 GMT
The only way to do something like this is get them mapers to use something 
like /dev/shm as there storage folder that's 100% memory
outside of that everything is flushed because the mapper exits when its done 
the tasktracker is the one delivering the output to the reduce task.

Billy



"paula_ta" <paula_ta@yahoo.com> wrote in 
message news:23617347.post@talk.nabble.com...
>
> Is it possible that some intermediate data produced by mappers and written 
> to
> the local file system resides in memory in the file system cache and is
> never flushed to disk ?  Eventually reducers will retrieve this data via
> HTTP - possibly without the data ever being written to disk ?
>
> thanks
> Paula
>
> -- 
> View this message in context: 
> http://www.nabble.com/Is-intermediate-data-produced-by-mappers-always-flushed-to-disk---tp23617347p23617347.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
> 



Mime
View raw message