hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ram <pramesh...@gmail.com>
Subject Re: copy files from ftp to hdfs in parallel, distcp failed
Date Tue, 16 Jul 2013 15:03:06 GMT
Hi,
    Please replace 0.0.0.0.with your ftp host ip address and try it.

Hi,



From,
Ramesh.




On Mon, Jul 15, 2013 at 3:22 PM, Hao Ren <h.ren@claravista.fr> wrote:

>  Thank you, Ram
>
> I have configured core-site.xml as following:
>
> <?xml version="1.0"?>
> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
>
> <!-- Put site-specific property overrides in this file. -->
>
> <configuration>
>
>     <property>
>         <name>hadoop.tmp.dir</name>
>         <value>/vol/persistent-hdfs</value>
>     </property>
>
>     <property>
>         <name>fs.default.name</name>
>         <value>hdfs://ec2-23-23-33-234.compute-1.amazonaws.com:9010
> </value>
>     </property>
>
>     <property>
>         <name>io.file.buffer.size</name>
>         <value>65536</value>
>     </property>
>
>     <property>
>         <name>fs.ftp.host</name>
>         <value>0.0.0.0</value>
>     </property>
>
>     <property>
>         <name>fs.ftp.host.port</name>
>         <value>21</value>
>     </property>
>
> </configuration>
>
> Then I tried  hadoop fs -ls file:/// , it works.
> But hadoop fs -ls ftp://<login>:<password>@<ftp server ip>/<directory>/
> doesn't work as usual:
>     ls: Cannot access ftp://<user>:<password>@<ftp server
> ip>/<directory>/: No such file or directory.
>
> When ignoring <directroy> as :
>
> hadoop fs -ls ftp://<login>:<password>@<ftp server ip>/
>
> There are no error msgs, but it lists nothing.
>
>
> I have also check the rights for my /home/<user> directroy:
>
> drwxr-xr-x 11 <user> <user>  4096 jui 11 16:30 <user>
>
> and all the files under /home/<user> have rights 755.
>
> I can easily copy the link ftp://<user>:<password>@<ftp server
> ip>/<directory>/ to firefox, it lists all the files as expected.
>
> Any workaround here ?
>
> Thank you.
>
> Le 12/07/2013 14:01, Ram a écrit :
>
> Please configure the following in core-ste.xml and try.
>    Use hadoop fs -ls file:///  -- to display local file system files
>    Use hadoop fs -ls ftp://<your ftp location>   -- to display ftp files
> if it is listing files go for distcp.
>
>  reference from
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/core-default.xml
>
>
>    fs.ftp.host 0.0.0.0 FTP filesystem connects to this server
> fs.ftp.host.port 21 FTP filesystem connects to fs.ftp.host on this port
>
>
>
> --
> Hao Ren
> ClaraVistawww.claravista.fr
>
>

Mime
View raw message