hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: What's the purpose of the variables numInFlight and numCopied?
Date Sun, 02 Jan 2011 13:44:09 GMT
Hello,

On Sun, Jan 2, 2011 at 5:42 PM, Pedro Costa <psdc1978@gmail.com> wrote:
> The reduce task contains in the method fetchOutputs, 2 variable:
> numInFlight
> numCopied
>
> What are the purpose of these variables?
>

Assuming the code of ReduceTask.java is from the 0.20 branch:
numInFlight is the count of all scheduled map output copy operations.
It is being used to determine if the output fetching process is busy
enough with several such operations.

numCopied is a counter that is incremented at every map output copy's
success, reaching its final successful value at the number of Maps
itself.

Something like "we have ten files to copy (numMaps), one has been
copied properly (numCopied) and five are in progress (numInFlight)."

-- 
Harsh J
www.harshj.com

Mime
View raw message