hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Momina Khan <momina.a...@gmail.com>
Subject plz help! HDFS to S3 copy issues
Date Wed, 11 Jul 2012 18:23:22 GMT
i use the following command to try to copy data from hdfs to my s3 bucket

ubuntu@domU-12-31-39-04-6E-58:
/state/partition1/hadoop-1.0.1$ *bin/hadoop distcp hdfs://
10.240.113.162:9001/data/ s3://ID:**SECRET@momin*a

java throws a connection refused exception ... i am running just this one
instance the same URI works fine for other hdfs commands ... even localhost
gives the same error ....
plz help i have also tried hftp but i am guessing connection refused is not
a distcp error ... have tried all i can ... i have the authentication
certificate private key in place ... could it be an authentication failure
... but the connection refused is mentioned on the hdfs URI.

plz anyone help ... have tried google extensively!*

*Find the call trace attached below:

ubuntu@domU-12-31-39-04-6E-58:
/state/partition1/hadoop-1.0.1$ *bin/hadoop distcp hdfs://
10.240.113.162:9001/data/ s3://ID:**SECRET@momina
*

12/07/05 12:48:37 INFO tools.DistCp: srcPaths=[hdfs://
10.240.113.162:9001/data]
12/07/05 12:48:37 INFO tools.DistCp: destPath=s3://ID:SECRET@momina

12/07/05 12:48:38 INFO ipc.Client: Retrying connect to server:
domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
tried 0 time(s).
12/07/05 12:48:39 INFO ipc.Client: Retrying connect to server:
domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
tried 1 time(s).
12/07/05 12:48:40 INFO ipc.Client: Retrying connect to server:
domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
tried 2 time(s).
12/07/05 12:48:41 INFO ipc.Client: Retrying connect to server:
domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
tried 3 time(s).
12/07/05 12:48:42 INFO ipc.Client: Retrying connect to server:
domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
tried 4 time(s).
12/07/05 12:48:43 INFO ipc.Client: Retrying connect to server:
domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
tried 5 time(s).
12/07/05 12:48:44 INFO ipc.Client: Retrying connect to server:
domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
tried 6 time(s).
12/07/05 12:48:45 INFO ipc.Client: Retrying connect to server:
domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
tried 7 time(s).
12/07/05 12:48:46 INFO ipc.Client: Retrying connect to server:
domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
tried 8 time(s).
12/07/05 12:48:47 INFO ipc.Client: Retrying connect to server:
domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001. Already
tried 9 time(s).
With failures, global counters are inaccurate; consider running with -i
Copy failed: java.net.ConnectException: Call to
domU-12-31-39-04-6E-58.compute-1.internal/10.240.113.162:9001 failed on
connection exception: java.net.ConnectException: Connection refused
    at org.apache.hadoop.ipc.Client.wrapException(Client.java:1095)
    at org.apache.hadoop.ipc.Client.call(Client.java:1071)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at $Proxy1.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
    at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:187)
    at org.apache.hadoop.tools.DistCp.checkSrcPath(DistCp.java:635)
    at org.apache.hadoop.tools.DistCp.copy(DistCp.java:656)
    at org.apache.hadoop.tools.DistCp.run(DistCp.java:881)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.hadoop.tools.DistCp.main(DistCp.java:908)
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
    at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
    at
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:434)
    at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:560)
    at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:184)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1202)
    at org.apache.hadoop.ipc.Client.call(Client.java:1046)
    ... 19 more

thank u!
momina

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message