hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: REG:error details(temporary directory does not exist)
Date Wed, 21 Dec 2011 11:53:39 GMT
Jayaseelan,

Am not sure if your NFS can take that much load, but do try with a
smaller dataset to see if it works. You'll need the destination path
available on all the MR nodes for distcp to succeed -- cause its
essentially going to try and write from each of a map task (distcp
runs a job).

And, copyToLocal will work for directories too - its not just for files.

On Wed, Dec 21, 2011 at 3:31 PM, Jayaseelan E <jayaseelan.e@ericsson.com> wrote:
> Hi Harsh
> basically i have some 40 folders each folder size is approximately 3GB .To
> move tis folder from HDFS to local location.I used the distcp..
> Iam not copying files, but iam copying the whole folders which contains
> folders and files.
> So i cannot use the copyToLocal command.
> May be i will try this distcp with NFS location
>
> thanks indeed
> jayaseelan
>
> ________________________________
> From: Harsh J [mailto:harsh@cloudera.com]
> Sent: Wednesday, December 21, 2011 3:26 PM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: REG:error details(temporary directory does not exist)
>
> Hey Jayaseelan,
>
> You can't distcp to a file:// destination URI, unless that location is a NFS
> point commonly available across all your nodes.
>
> What you probably need for 3 GB worth of data is a simple "hadoop fs
> -copyToLocal src dest" command.
>
> On 21-Dec-2011, at 12:20 PM, Jayaseelan E wrote:
>
>
> Hi Folks!
> I am have some 40 folders whose size of each folders are aproximately 3GB.I
> used distcp command to take a copy from HDFS to local machine.
> But it fails with the following error.
> /opt/hadoop/bin/hadoop distcp   hdfs://bl460x1890:9000/<<Source
> location>>  file:///<<targetpath>>
>
> java.io.IOException: The temporary job-output
> directory file:/archive/_distcp_logs_ndyr8m/_temporarydoesn't exist!
>         at
> org.apache.hadoop.mapred.FileOutputCommitter.getWorkPath(FileOutputCommitter.java:204)
>         at
> org.apache.hadoop.mapred.FileOutputFormat.getTaskOutputPath(FileOutputFormat.java:234)
>
> My hadoop version is 2.0.1
> If any one can please help me resolve this
>
> Thanks indeed
> jayaseelan
>
>
>



-- 
Harsh J

Mime
View raw message