hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris K Wensel <ch...@wensel.net>
Subject Re: hadoop on EC2
Date Wed, 04 Jun 2008 15:31:10 GMT

These are the FoxyProxy wildcards I use

*compute-1.amazonaws.com*
*.ec2.internal*
*.compute-1.internal*

and w/ hadoop 0.17.0, just type (after booting your cluster)

hadoop-ec2 proxy <cluster-name>

to start the tunnel for that cluster

On Jun 3, 2008, at 11:26 PM, James Moore wrote:

> On Tue, Jun 3, 2008 at 5:04 PM, Andreas Kostyrka  
> <andreas@kostyrka.org> wrote:
>> Plus to make it even more painful, you cannot easily run it with  
>> one simple
>> SOCKS server, because you need to defer DNS resolution to the  
>> inside the
>> cluster, because VM names do resolve to external IPs, while the  
>> webservers
>> we'd be all interested in reside on the internal 10/8 IPs.
>
> It's easy with foxyproxy.
>
> Run ssh with the -D command:
>
> ssh -D 2324 ec2-75-101-XXX-XX.compute-1.amazonaws.com
>
> Tell FoxyProxy to "use SOCKS proxy for DNS lookups" (tools > foxyproxy
>> more > global settings > use SOCKS proxy for DNS lookups)
>
> Configure foxyproxy with rules for when to use local port 2324.  Use
> wildcards like http*ec2*internal*.  I put a screenshot on my blog -
> http://blog.restphone.com/2008/6/4/foxyproxy-hadoop-and-socks
>
> All the features I cared about worked when set up this way.
>
> (And of course the choice of 2324 isn't special - use any port you  
> like.)
> -- 
> James Moore | james@restphone.com
> Ruby and Ruby on Rails consulting
> blog.restphone.com

Chris K Wensel
chris@wensel.net
http://chris.wensel.net/
http://www.cascading.org/





Mime
View raw message