hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Owen O'Malley <o...@yahoo-inc.com>
Subject Re: What is a spill to disk?
Date Thu, 11 Jan 2007 23:43:13 GMT

On Jan 11, 2007, at 1:06 PM, Dennis Kubes wrote:

> In reading through some entries on the dev and commit lists I keep  
> seeing talk about spills to disk? Can someone explain what that is?

In a couple of places data is stored until a buffer is full, and then  
the entire buffer is processed and written to disk. One such place is  
the output from the maps. The data is kept in memory, and when the  
buffer is full, it is sorted and written to disk. We call the  
flushing of the buffer "spilling".

-- Owen

View raw message