hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tim robertson <timrobertson...@gmail.com>
Subject Re: Help with EC2 using HDFS client (port open issue?)
Date Sat, 20 Jun 2009 18:48:30 GMT
Thanks Harish

Turned out I was mistakenly using the wrong port altogether

Cheers

Tim


On Sat, Jun 20, 2009 at 6:40 PM, Harish
Mallipeddi<harish.mallipeddi@gmail.com> wrote:
> Hi Tim,
>
> I don't know the answer to your specific problem but IIRC all ports on EC2
> machines (within the same security group?) are open and reachable within the
> EC2 environment. You only have to open the ports (via ec2-authorize) if you
> want to reach them from outside EC2. So typically for hadoop clusters, you
> open the ports corresponding to the web-admin consoles (ports 50030, 50060,
> 50070, etc) so you can see the web console from your browser. I've not used
> EC2 for a year now so things might have changed.
>
> Also do use the 'public' hostnames for configuring purposes - they resolve
> to internal IPs inside EC2 and external IPs from outside EC2.
>
> More on this:
> http://mail-archives.apache.org/mod_mbox/hadoop-core-user/200905.mbox/%3CDFD95197F3AE8C45B0A96C2F4BA3A2556BF123EE37@SC-MBXC1.TheFacebook.com%3E
>
> - Harish
>
> On Sat, Jun 20, 2009 at 6:12 PM, tim robertson <timrobertson100@gmail.com>wrote:
>
>> Hi all,
>>
>> I am using Hadoop to build a read only store for voldemort on EC2 and
>> for some reason can't get it to talk across the nodes.
>> I know this is a specific EC2 linux setup question, but I was hoping
>> someone could help me as I am sure all the apps build on Hadoop are
>> doing this - I'm not very hot on linux.
>>
>> The client is calling
>>
>> hdfs://ip-10-244-191-175.ec2.internal:54310/user/root/output/fullPD/stage2/node-0
>>
>> and I have run
>>  ec2-authorize hdfs-cluster -p 54310
>> (but I am not sure this is the way to open the port)
>>
>> I'm using the cloudera AMI.
>> Full trace is below and any pointers are greatly appreciated!
>>
>> Cheers
>>
>> Tim
>>
>>
>> 09/06/20 06:54:09 ERROR gui.ReadOnlyStoreManagementServlet: Error
>> while performing operation.
>> java.net.ConnectException: Call to
>> ip-10-244-191-175.ec2.internal/10.244.191.175:54310 failed on
>> connection exception: java.net.ConnectException: Connection refused
>>        at org.apache.hadoop.ipc.Client.wrapException(Client.java:743)
>>        at org.apache.hadoop.ipc.Client.call(Client.java:719)
>>        at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:216)
>>        at org.apache.hadoop.dfs.$Proxy6.getProtocolVersion(Unknown Source)
>>        at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:348)
>>        at
>> org.apache.hadoop.dfs.DFSClient.createRPCNamenode(DFSClient.java:103)
>>        at org.apache.hadoop.dfs.DFSClient.<init>(DFSClient.java:172)
>>        at
>> org.apache.hadoop.dfs.DistributedFileSystem.initialize(DistributedFileSystem.java:67)
>>        at
>> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1328)
>>        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:56)
>>        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1343)
>>        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:213)
>>        at org.apache.hadoop.fs.Path.getFileSystem(Path.java:175)
>>        at
>> voldemort.store.readonly.fetcher.HdfsFetcher.fetch(HdfsFetcher.java:82)
>>        at
>> voldemort.server.http.gui.ReadOnlyStoreManagementServlet.doFetch(ReadOnlyStoreManagementServlet.java:162)
>>        at
>> voldemort.server.http.gui.ReadOnlyStoreManagementServlet.doPost(ReadOnlyStoreManagementServlet.java:125)
>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
>>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>>        at
>> org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:502)
>>        at
>> org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:389)
>>        at
>> org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:765)
>>        at
>> org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
>>        at org.mortbay.jetty.Server.handle(Server.java:326)
>>        at
>> org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:534)
>>        at
>> org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:879)
>>        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:747)
>>        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>>        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>>        at
>> org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:409)
>>        at
>> org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:520)
>>
>
>
>
> --
> Harish Mallipeddi
> http://blog.poundbang.in
>

Mime
View raw message