whirr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (WHIRR-612) CDH4 can be installed on Ubuntu now as well as CentOS
Date Wed, 15 Aug 2012 17:51:38 GMT

    [ https://issues.apache.org/jira/browse/WHIRR-612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13435346#comment-13435346
] 

Tom White commented on WHIRR-612:
---------------------------------

The YARN test is failing on Rackspace with 

{noformat}
2012-08-15 12:25:24,529 INFO  ipc.Client (Client.java:handleConnectionFailure(683)) - Retrying
connect to server: 67-207-153-65.static.cloud-ips.com/67.207.153.65:8040. Already tried 9
time(s).
java.lang.reflect.UndeclaredThrowableException
	at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:135)
	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:134)
	at org.apache.hadoop.mapred.ResourceMgrDelegate.getNewJobID(ResourceMgrDelegate.java:181)
	at org.apache.hadoop.mapred.YARNRunner.getNewJobID(YARNRunner.java:214)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:338)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1226)
	at org.apache.hadoop.mapreduce.Job$11.run(Job.java:1223)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1223)
	at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1244)
	at org.apache.hadoop.examples.WordCount.main(WordCount.java:84)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
	at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
	at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:68)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: com.google.protobuf.ServiceException: java.io.IOException: Failed on local exception:
java.net.SocketException: Malformed reply from SOCKS server; Host Details : local host is:
"Clouderas-MacBook-Pro-3.local/192.168.0.188"; destination host is: "67-207-153-65.static.cloud-ips.com":8040;

	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:187)
	at $Proxy10.getNewApplication(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:132)
	... 23 more
Caused by: java.io.IOException: Failed on local exception: java.net.SocketException: Malformed
reply from SOCKS server; Host Details : local host is: "Clouderas-MacBook-Pro-3.local/192.168.0.188";
destination host is: "67-207-153-65.static.cloud-ips.com":8040; 
	at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:765)
	at org.apache.hadoop.ipc.Client.call(Client.java:1165)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
	... 25 more
Caused by: java.net.SocketException: Malformed reply from SOCKS server
	at java.net.SocksSocketImpl.readSocksReply(SocksSocketImpl.java:147)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:538)
	at java.net.Socket.connect(Socket.java:529)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:522)
	at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:489)
	at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:472)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:566)
	at org.apache.hadoop.ipc.Client$Connection.access$2000(Client.java:215)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1271)
	at org.apache.hadoop.ipc.Client.call(Client.java:1141)
	... 26 more
{noformat}

Port 8040 is the Reource Manager address, which the client talks to. The problem is that the
client tries to talk to the public hostname (e.g. 67-207-153-65.static.cloud-ips.com) which
is resolved on the head node (over the SSH SOCKS tunnel) to the public IP address 67.207.153.65.
However, the localizer is only listening on the private address so we get a connection refused.

In early versions of YARN the RM would listen on all interfaces, however MAPREDUCE-4163 changed
the behaviour to listen on a single interface.

I think this works on AWS since the public hostname is resolved to the private IP since the
resolution happens on the cluster. Rackspace doesn't have this behaviour, so it fails.

Since this is a limitation in YARN (and it cannot be overidden as far as I can see) I think
we should ship 0.8.0 with this known issue, while we work out how to get YARN to work on Rackspace
for a later release. (This will probably require fixes in YARN and Whirr.) We should commit
this patch along with the whirr.env.MAPREDUCE_VERSION thing. Does that sound reasonable?


                
> CDH4 can be installed on Ubuntu now as well as CentOS
> -----------------------------------------------------
>
>                 Key: WHIRR-612
>                 URL: https://issues.apache.org/jira/browse/WHIRR-612
>             Project: Whirr
>          Issue Type: Bug
>          Components: service/cdh
>    Affects Versions: 0.8.0
>            Reporter: Andrew Bayer
>            Assignee: Andrew Bayer
>             Fix For: 0.8.0
>
>         Attachments: cdh-yarn-cloudservers-us.txt, cdh-yarn-rackspace-cloudservers-us.txt,
WHIRR-612.patch, WHIRR-612.patch, WHIRR-612.patch
>
>
> CDH4 beta 1 was only available on CentOS, but from beta 2 onward, CDH4 has been available
on Ubuntu et al. So we should remove the hardcoding in tests and recipes for centos.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message