hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron Kimball" <aa...@cloudera.com>
Subject Re: distcp port for 0.17.2
Date Thu, 23 Oct 2008 09:38:00 GMT
Hi,

The dfs.http.address is for human use, not program interoperability. You can
visit http://whatever.address.your.namenode.has:50070 in a web browser and
see statistics about your filesystem.

The address of cluster 2 is in its fs.default.name. This should be set to
something like hdfs://cluster2.master.name:9000/

The file:// "protocol" only refers to paths on the current machine in its
real (non-DFS) filesystem.
- Aaron

On Wed, Oct 22, 2008 at 3:47 PM, bzheng <bing.zheng@gmail.com> wrote:

>
> Thanks.  The fs.default.name is "file:///" and dfs.http.address is
> "0.0.0.0:50070".  I tried:
>
> hadoop dfs -ls /path/file to make sure file exists on cluster1
> hadoop distcp file:///cluster1_master_node_ip:50070/path/file
> file:///cluster2_master_node_ip:50070/path/file
>
> It gives this error message:
> 08/10/22 15:43:47 INFO util.CopyFiles:
> srcPaths=[file:/cluster1_master_node_ip:50070/path/file]
> 08/10/22 15:43:47 INFO util.CopyFiles:
> destPath=file:/cluster2_master_node_ip:50070/path/file
> With failures, global counters are inaccurate; consider running with -i
> Copy failed: org.apache.hadoop.mapred.InvalidInputException: Input source
> file:/cluster1_master_node_ip:50070/path/file does not exist.
>        at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:578)
>        at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594)
>        at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>        at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763)
>
>
> If I use hdfs:// instead of file:///, I get:
> Copy failed: java.net.SocketTimeoutException: timed out waiting for rpc
> response
>        at org.apache.hadoop.ipc.Client.call(Client.java:559)
>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>        at org.apache.hadoop.dfs.$Proxy0.getProtocolVersion(Unknown Source)
>        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:313)
>        at
> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:102)
>        at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:178)
>        at
>
> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:68)
>        at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1280)
>        at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:56)
>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1291)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:203)
>        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
>        at org.apache.hadoop.util.CopyFiles.checkSrcPath(CopyFiles.java:572)
>        at org.apache.hadoop.util.CopyFiles.copy(CopyFiles.java:594)
>        at org.apache.hadoop.util.CopyFiles.run(CopyFiles.java:743)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
>        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
>        at org.apache.hadoop.util.CopyFiles.main(CopyFiles.java:763)
>
>
>
> s29752-hadoopuser wrote:
> >
> > Hi,
> >
> > There is no such thing called distcp port.  distcp uses (generic) file
> > system API and so it does not care about the file system implementation
> > details like port number.
> >
> > It is common to use distcp with HDFS or HFTP.  The urls will look like
> > hdfs://namenode:port/path and hftp://namenode:port/path for HDFS and
> HFTP,
> > respectively.   The HDFS and HFTP ports are specified by fs.default.name
> > and dfs.http.address, respectively.
> >
> > Nicholas Sze
> >
> >
> >
> >
> > ----- Original Message ----
> >> From: bzheng <bing.zheng@gmail.com>
> >> To: core-user@hadoop.apache.org
> >> Sent: Wednesday, October 22, 2008 11:57:43 AM
> >> Subject: distcp port for 0.17.2
> >>
> >>
> >> What's the port number for distcp in 0.17.2?  I can't find any
> >> documentation
> >> on distcp for version 0.17.2.  For version 0.18, the documentation says
> >> it's
> >> 8020.
> >>
> >> I'm using a standard install and the only open ports associated with
> >> hadoop
> >> are 50030, 50070, and 50090.  None of them work with distcp.  So, how do
> >> you
> >> use distcp in 0.17.2?  are there any extra setup/configuration needed?
> >>
> >> Thanks in advance for your help.
> >> --
> >> View this message in context:
> >> http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20117463.html
> >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/distcp-port-for-0.17.2-tp20117463p20121246.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message