hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mischa Tuffield <mis...@mmt.me.uk>
Subject Re: copyToLocal Failed inside the cleanup(.........) of Map task
Date Mon, 03 Jun 2013 21:40:44 GMT
When you are configuring your job in what most people refer to as their "Driver" class you
can simply not set a Reducer and only set a Mapper. 

 // makes the job a map only one 
 job.setNumReduceTasks(0);

 job.setMapperClass(MyFooMapper.class);

Mischa

On 3 Jun 2013, at 22:11, samir das mohapatra <samir.helpdoc@gmail.com> wrote:

> Dear All,
> 
>      Is there any way to copy the intermediate output file of the mapper into     local
 folder  after each map task complete.
> 
>    Right now I am using 
>     
>    FileSystem.copyToLocalFile(hdfsLocation,localLocation); 
>  indiste the cleanup of mapper task , but it is failing . 
> 
> Exception file not found. 
> 
> But if I am giving same statement after the job complete in driver class ,it is working
fine. that i dont want.
> 
> protected void cleanup(Context context){
>    FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
> 
> Note: I need to copy the inter mediate output of the mapper to local file system just
after  each map task complete. I dont want any reducer.
> 
>    If this is not the right solution then how to solve this type of scenario.
>   
> Any help.............
> 
> regards,
> samir.

_______________________________
Mischa Tuffield PhD
http://mmt.me.uk/
@mischat






Mime
View raw message