Mailing-List: contact hadoop-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hadoop-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of tom.e.white@gmail.com
 designates 64.233.166.179 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references;
        b=F9nJJS4zPEp0yZpMNOMXqhfmh4hPMKr0PWPcHdv2AvLdDZryY60977X0ziKYhHM0oAFJk64UICB4wuSo5QE4AzrDvmBrFJs3aA53Di0JJkk1uKr1fSRPZ9rJZ1M07qAaq/Zj/DIf3UTTjNJOle18dqFTONbtwuc213rqKSfnlPo=
Message-ID: <b23e01b50709060132y35cd60fcjd7ad9c430097d654@mail.gmail.com>
Date: Thu, 6 Sep 2007 09:32:49 +0100
From: "Tom White" <tom.e.white@gmail.com>
To: hadoop-user@lucene.apache.org
Subject: Re: Accessing S3 with Hadoop?
In-Reply-To: <876ef97a0709051539p6003easbd93aaa1445f51a3@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: <876ef97a0709051219o7323cd60u99be07d4896e91c4@mail.gmail.com>
	 <38e0289d0709051336v4f97aefxbc0b2d03980af8bf@mail.gmail.com>
	 <b23e01b50709051501y7cac9e0dj5cd2bd52dfc78245@mail.gmail.com>
	 <876ef97a0709051539p6003easbd93aaa1445f51a3@mail.gmail.com>

> Yeah, I actually read all of the wiki and your article about using
> Hadoop on EC2/S3 and I can't really find a reference to the S3 support
> not being for "regular" S3 keys. Did I miss something or should I
> update the wiki to make it more clear (or both)?

I don't think this is explained clearly enough, so please do update
the wiki. Thanks.

> Also, the instructions on the EC2 page on the wiki no longer work, in
> that due to the kind of NAT Amazon is using, the slaves can't connect
> to the master using an externally-resolved IP address via a DNS name.
> What I mean is, if you set DNS to the external IP of your master
> instance, your slaves can resolve that address but cannot then connect
> to it. So, I had to alter the launch-hadoop-cluster and start-hadoop
> scripts and merge them to just pick the master and use its EC2-given
> name as the $MASTER_HOST to make it work.

This sounds like the problem fixed in
https://issues.apache.org/jira/browse/HADOOP-1638 in 0.14.0, which is
the version you're using isn't it?

Are you able to do 'bin/hadoop-ec2 launch-cluster' then (on your workstation)

. bin/hadoop-ec2-env.sh
ssh $SSH_OPTS "root@$MASTER_HOST" "sed -i -e
\"s/$MASTER_HOST/\$(hostname)/g\"
/usr/local/hadoop-$HADOOP_VERSION/conf/hadoop-site.xml"

and then check to see if the master host has been set correctly (to
the internal IP) in the master host's hadoop-site.xml.

Also, what version of the EC2 tools are you using?

> I also updated the scripts
> to only look for a given AMI ID and only start/manage/terminate
> instances of that AMI ID (since I have others I'd rather not
> terminated just on the basis of their AMI launch index ;-)).

Instances are terminated on the basis of their AMI ID since 0.14.0.
See https://issues.apache.org/jira/browse/HADOOP-1504.

Tom