hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raj K Singh <rajkrrsi...@gmail.com>
Subject Re: copyToLocal Failed inside the cleanup(.........) of Map task
Date Tue, 04 Jun 2013 04:54:43 GMT
by default hadoop keep intermediate values produced by mapper in the local
file system,you can get the the handle on it using
FileOutputFormat.getWorkOutputPath(context)

::::::::::::::::::::::::::::::::::::::::
Raj K Singh
http://www.rajkrrsingh.blogspot.com
Mobile  Tel: +91 (0)9899821370


On Tue, Jun 4, 2013 at 2:41 AM, samir das mohapatra <samir.helpdoc@gmail.com
> wrote:

> Dear All,
>
>      Is there any way to copy the intermediate output file of the mapper
> into     local  folder  after each map task complete.
>
>    Right now I am using
>
>    FileSystem.copyToLocalFile(hdfsLocation,localLocation);
>  indiste the cleanup of mapper task , but it is failing .
>
> Exception file not found.
>
> But if I am giving same statement after the job complete in driver class
> ,it is working fine. that i dont want.
>
> protected void cleanup(Context context){
>    FileSystem.copyToLocalFile(hdfsLocation,localLocation);//failed
> }
>
> Note: I need to copy the inter mediate output of the mapper to local file
> system just after  each map task complete. I dont want any reducer.
>
>    If this is not the right solution then how to solve this type of
> scenario.
>
> Any help.............
>
> regards,
> samir.
>

Mime
View raw message