hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Using S3 instead of HDFS
Date Wed, 18 Jan 2012 06:52:03 GMT
When using S3 you do not need to run any component of HDFS at all. It
is meant to be an alternate FS choice. You need to run only MR.

The wiki page at http://wiki.apache.org/hadoop/AmazonS3 mentions on
how to go about specifying your auth details to S3, either directly
via the fs.default.name URI or via the additional properties
fs.s3.awsAccessKeyId and fs.s3.awsSecretAccessKey. Does this not work
for you?

On Wed, Jan 18, 2012 at 12:14 PM, Mark Kerzner <mark.kerzner@shmsoft.com> wrote:
> Well, here is my error message
>
> Starting Hadoop namenode daemon: starting namenode, logging to
> /usr/lib/hadoop-0.20/logs/hadoop-hadoop-namenode-ip-10-126-11-26.out
> ERROR. Could not start Hadoop namenode daemon
> Starting Hadoop secondarynamenode daemon: starting secondarynamenode,
> logging to
> /usr/lib/hadoop-0.20/logs/hadoop-hadoop-secondarynamenode-ip-10-126-11-26.out
> Exception in thread "main" java.lang.IllegalArgumentException: Invalid URI
> for NameNode address (check fs.default.name): s3n://myname.testdata is not
> of scheme 'hdfs'.
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:224)
>        at
> org.apache.hadoop.hdfs.server.namenode.NameNode.getServiceAddress(NameNode.java:209)
>        at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.initialize(SecondaryNameNode.java:182)
>        at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.<init>(SecondaryNameNode.java:150)
>        at
> org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode.main(SecondaryNameNode.java:624)
> ERROR. Could not start Hadoop secondarynamenode daemon
>
> And, if I don't need to start the NameNode, then where do I give the S3
> credentials?
>
> Thank you,
> Mark
>
>
> On Wed, Jan 18, 2012 at 12:36 AM, Harsh J <harsh@cloudera.com> wrote:
>
>> Hey Mark,
>>
>> What is the exact trouble you run into? What do the error messages
>> indicate?
>>
>> This should be definitive enough I think:
>> http://wiki.apache.org/hadoop/AmazonS3
>>
>> On Wed, Jan 18, 2012 at 11:55 AM, Mark Kerzner <mark.kerzner@shmsoft.com>
>> wrote:
>> > Hi,
>> >
>> > whatever I do, I can't make it work, that is, I cannot use
>> >
>> > s3://host
>> >
>> > or s3n://host
>> >
>> > as a replacement for HDFS while runnings EC2 cluster. I change the
>> settings
>> > in the core-file.xml, in hdfs-site.xml, and start hadoop services, and it
>> > fails with error messages.
>> >
>> > Is there a place where this is clearly described?
>> >
>> > Thank you so much.
>> >
>> > Mark
>>
>>
>>
>> --
>> Harsh J
>> Customer Ops. Engineer, Cloudera
>>



-- 
Harsh J
Customer Ops. Engineer, Cloudera

Mime
View raw message