hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Vyas <jayunit...@gmail.com>
Subject Re: Writing intermediate key,value pairs to file and read it again
Date Sat, 20 Apr 2013 16:06:18 GMT
How many intermediate keys? If small enough, you can keep them in memory.  If large, you can
just wait for the job to finish and siphon them into your job as input with the MultipleInputs
API.



On Apr 20, 2013, at 10:43 AM, Vikas Jadhav <vikascjadhav87@gmail.com> wrote:

> Hello,
> Can anyone help me in following issue
> Writing intermediate key,value pairs to file and read it again
> 
> let us say i have to write each intermediate pair received @reducer to a file and again
read that as key value pair again and use it for processing
> 
> I found IFile.java file which has reader and writer but i am not able understand how
to use it for example. I dont understand Counter value as last parameter "spilledRecordsCounter"

> 
> 
> Thanks.
> 
> 
> -- 
> 
> 
>   Regards,
>    Vikas

Mime
View raw message