hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harish Mallipeddi <harish.mallipe...@gmail.com>
Subject Re: Help with EC2 using HDFS client (port open issue?)
Date Sat, 20 Jun 2009 16:40:09 GMT
Hi Tim,

I don't know the answer to your specific problem but IIRC all ports on EC2
machines (within the same security group?) are open and reachable within the
EC2 environment. You only have to open the ports (via ec2-authorize) if you
want to reach them from outside EC2. So typically for hadoop clusters, you
open the ports corresponding to the web-admin consoles (ports 50030, 50060,
50070, etc) so you can see the web console from your browser. I've not used
EC2 for a year now so things might have changed.

Also do use the 'public' hostnames for configuring purposes - they resolve
to internal IPs inside EC2 and external IPs from outside EC2.

More on this:
http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200905.mbox/%3CDFD95197F3AE8C45B0A96C2F4BA3A2556BF123EE37@SC-MBXC1.TheFacebook.com%3E

- Harish

On Sat, Jun 20, 2009 at 6:12 PM, tim robertson <timrobertson100@gmail.com>wrote:

> Hi all,
>
> I am using Hadoop to build a read only store for voldemort on EC2 and
> for some reason can't get it to talk across the nodes.
> I know this is a specific EC2 linux setup question, but I was hoping
> someone could help me as I am sure all the apps build on Hadoop are
> doing this - I'm not very hot on linux.
>
> The client is calling
>
> hdfs://ip-10-244-191-175.ec2.internal:54310/user/root/output/fullPD/stage2/node-0
>
> and I have run
>  ec2-authorize hdfs-cluster -p 54310
> (but I am not sure this is the way to open the port)
>
> I'm using the cloudera AMI.
> Full trace is below and any pointers are greatly appreciated!
>
> Cheers
>
> Tim
>
>
> 09/06/20 06:54:09 ERROR gui.ReadOnlyStoreManagementServlet: Error
> while performing operation.
> java.net.ConnectException: Call to
> ip-10-244-191-175.ec2.internal/10.244.191.175:54310 failed on
> connection exception: java.net.ConnectException: Connection refused
>        at org.apache.hadoop.ipc.Client.wrapException(Client.java:743)
>        at org.apache.hadoop.ipc.Client.call(Client.java:719)
>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>        at org.apache.hadoop.dfs.$Proxy6.getProtocolVersion(Unknown Source)
>        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
>        at
> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:103)
>        at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:172)
>        at
> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:67)
>        at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
>        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:56)
>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1343)
>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)
>        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
>        at
> voldemort.store.readonly.fetcher.HdfsFetcher.fetch(HdfsFetcher.java:82)
>        at
> voldemort.server.http.gui.ReadOnlyStoreManagementServlet.doFetch(ReadOnlyStoreManagementServlet.java:162)
>        at
> voldemort.server.http.gui.ReadOnlyStoreManagementServlet.doPost(ReadOnlyStoreManagementServlet.java:125)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>        at
> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>        at
> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
>        at
> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
>        at
> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>        at org.mortbay.jetty.Server.handle(Server.java:326)
>        at
> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>        at
> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
>        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
>        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>        at
> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>        at
> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
>



-- 
Harish Mallipeddi
http://blog.poundbang.in

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message