hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alok Kumar <alok...@gmail.com>
Subject problem configuring hadoop with s3 bucket
Date Mon, 23 Jul 2012 08:26:21 GMT
Hello Group,

I've hadoop setup locally running.

Now I want to use Amazon s3://<mybucket> as my data store,
so i changed like " dfs.data.dir=s3://<mybucket>/hadoop/ " in my
hdfs-site.xml, Is it the correct way?
I'm getting error :

WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Invalid directory in
dfs.data.dir: can not create directory: s3://<mybucket>/hadoop
2012-07-23 13:15:06,260 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: All directories in
dfs.data.dir are invalid.

and
when i changed like " dfs.data.dir=s3://<mybucket>/ "
I got error :
 ERROR org.apache.hadoop.hdfs.server.datanode.DataNode:
java.lang.IllegalArgumentException: Wrong FS: s3://<mybucket>/, expected:
file:///
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
    at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
    at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
    at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
    at
org.apache.hadoop.util.DiskChecker.mkdirsWithExistsAndPermissionCheck(DiskChecker.java:146)
    at org.apache.hadoop.util.DiskChecker.checkDir(DiskChecker.java:162)
    at
org.apache.hadoop.hdfs.server.datanode.DataNode.makeInstance(DataNode.java:1574)
    at
org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:1521)
    at
org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:1539)
    at
org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:1665)
    at
org.apache.hadoop.hdfs.server.datanode.DataNode.main(DataNode.java:1682)

Also,
When I'm changing fs.default.name=s3://<<mybucket> , Namenode is not coming
up with error : ERROR org.apache.hadoop.hdfs.server.namenode.NameNode:
java.net.BindException: (Any way I want to run namenode locally, so I
reverted it back to hdfs://localhost:9000 )

Your help is highly appreciated!
Thanks
-- 
Alok Kumar

Mime
View raw message