hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alexander Aristov" <alexander.aris...@gmail.com>
Subject Re: Getting Hadoop Working on EC2/S3
Date Tue, 30 Sep 2008 06:23:15 GMT
Does you AWS (S3) key contain the "?" sing ? If so then it can be a cause.
Regenerate the key in this case.

I have also tried to use the create-image command but I stopped all attempts
after constant failures, It was easier to make AMI by hands.

Alexander

2008/9/29 Stephen Watt <swatt@us.ibm.com>

> Hi Folks
>
> Before I get started, I just want to state that I've done the due
> diligence and read Tom White's articles as well as EC2 and S3 pages on the
> Hadoop Wiki and done some searching on this.
>
> Thus far I have successfully got Hadoop running on EC2 with no problems.
> In my local hadoop 0.18 environment I simply add my AWS keys to the
> hadoop-ec2-env.sh and kickoff the src/contrib/ec2/bin/hadoop-ec2 launch
> cluster script and it works great.
>
> Now, I'm trying to use the Public Haodop EC2 images to run over S3 instead
> of HDFS. They are set up to use variables passed in from a parameterized
> launch for all the config options everything EXCEPT the
> fs.default.filesystem. So in order to bring a cluster of 20 hadoop
> instances up that run over S3, I need to mod the config file to point to
> my S3 bucket for the fs.default.filesystem and keep the rest the same.
> Thus I need my own image to do this.  I am attempting this by using the
> local src/contrib/ec2/bin/hadoop-ec2 create-image script. I've tried this
> both on a windows system (cygwin environment) AND on my ubuntu 8 system
> and with each one it gets all the way to the end and fails as it attempts
> to save the new image to my bucket and says the bucket does not exist with
> a Server.NoSuchBucket (404) error.
>
> The S3 bucket definitely does exist. I have block data inside of it that
> are results of my Hadoop Jobs. I can go to a single hadoop image on EC2
> that I've launched and manually set up to use S3 and say bin/hadoop dfs
> -ls / and I can see the contents of my S3 bucket. I can also succesfully
> use that s3 bucket as an input and output of my jobs for a single EC2
> hadoop instance. I've tried creating new buckets using the FireFox S3
> Organizer plugin and specifying the scripts to save my new image to those
> and its still the same error.
>
> Any ideas ? Is anyone having similar problems ?
>
> Regards
> Steve Watt




-- 
Best Regards
Alexander Aristov

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message