hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deepak Vohra <dvohr...@yahoo.com>
Subject Re: copyToLocal Failed inside the cleanup(.........) of Map task
Date Mon, 03 Jun 2013 21:57:31 GMT
Samir,

The intermediate output of the mapper is already output to the local filesystem, not HDFS. 

The temporary intermediate file path is 

FileOutputFormat.getWorkOutputPath(context)




________________________________
 From: samir das mohapatra <samir.helpdoc@gmail.com>
To: user@hadoop.apache.org; user-help@hadoop.apache.org; cdh-user@cloudera.org; cdh-user@cloudera.com

Sent: Monday, June 3, 2013 2:11 PM
Subject: copyToLocal Failed inside the cleanup(.........) of Map task
 


Dear All,

     Is there any way to copy the intermediate output file of the mapper into    
local  folder  after each map task complete.


   Right now I am using 
    
   FileSystem.copyToLocalFile(hdfsLocation,localLocation); 
 indiste the cleanup of mapper task , but it is failing . 

Exception file not found. 

But if I am giving same statement after the job complete in driver class ,it is working fine.
that i dont want.

protected void cleanup(Context context){
  FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
}


Note: I need to copy the inter mediate output of the mapper to local file system just after 
each map task complete. I dont want any reducer.


   If this is not the right solution then how to solve this type of scenario.

  

Any help.............


regards,
samir.
Mime
View raw message