incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Mackrory (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (S4-25) Write S4 Application Master to deploy S4 in Yarn
Date Tue, 04 Sep 2012 04:21:08 GMT

     [ https://issues.apache.org/jira/browse/S4-25?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Sean Mackrory updated S4-25:
----------------------------

    Attachment: S4-YARN-1.patch

I have attached a first attempt, although it is still a work in progress. I expect to have
more free time to further complete this over the next couple of weeks, but I wanted to share
my progress so far. I will replace the attachment with new patches as I add functionality.

This is based very closely on the distributedshell example shipped with YARN 2.0.1. There
are a lot of changes that ought to be made (removing code that's only there to demonstrate
features, documentation, etc.), but I am waiting until it's functional so I can make a minimal
diff of the changes required for S4 actions (which I suggest doing if you're going to look
at this code).

Feedback is obviously welcome - as this is my first work with Gradle, YARN or S4. I've filled
out the ApplicationMaster with what I think should be there, although it is completely untested
(it does compile) and is probably wrong. I am running Hadoop / YARN in pseudo-distributed
mode, and am attempting to deploy the app on the "cluster" with the following command:

{noformat}
./s4 yarn \
    -jar `pwd`/subprojects/s4-tools/build/libs/s4-tools-0.5.0-incubating.jar \
    -cluster=cluster1 \
    -nbTasks=2 \
    -flp=12000 \
    -s4r=`pwd`/build/libs/myApp.s4r
{noformat}

Currently it errors out when the Client communicates with the ResourceManager (it first occurs
in some of the unncessary demo code, but will eventually occur anyway). The corresponding
log message on the Client side is: 

{noformat}
584 [main] FATAL org.apache.s4.tools.yarn.Client  - Error running CLient
java.lang.reflect.UndeclaredThrowableException
	at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128)
	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getClusterMetrics(ClientRMProtocolPBClientImpl.java:123)
	at org.apache.s4.tools.yarn.Client.run(Client.java:331)
	at org.apache.s4.tools.yarn.Client.main(Client.java:184)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.apache.s4.tools.Tools$Task.dispatch(Tools.java:55)
	at org.apache.s4.tools.Tools.main(Tools.java:95)
Caused by: java.io.IOException: Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:371)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:409)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686)

	at org.apache.hadoop.ipc.Client.call(Client.java:1161)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:184)
	at $Proxy7.getClusterMetrics(Unknown Source)
	at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getClusterMetrics(ClientRMProtocolPBClientImpl.java:121)
	... 8 more
{noformat}

And the corresponding log message on the ResourceManager side is:

{noformat}
lPB.getClusterNodes from 127.0.1.1:51347: error: java.io.IOException: Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB
java.io.IOException: Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:371)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:409)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:916)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1692)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1688)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:396)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1686)
{noformat}

Please let me know if you have any feedback or hints on the "Unknown protocol" problem.
                
> Write S4 Application Master to deploy S4 in Yarn
> ------------------------------------------------
>
>                 Key: S4-25
>                 URL: https://issues.apache.org/jira/browse/S4-25
>             Project: Apache S4
>          Issue Type: New Feature
>            Reporter: J Mohamed Zahoor
>             Fix For: 0.6
>
>         Attachments: S4-YARN-1.patch
>
>
> On the lines of s4PigWrapper, write a s4 application master to host s4 piper inside Hadoop
Yarn. This could be useful not only for reading data stored in hadoop ( to build or train
a model)... But we could make use of the resource manager to deploy s4 instances in remote
machine and monitor them. In short, we could make use of most of the resource management ,
scheduling and other good stuff in Yarn.
> - Yarn is useful to deploy and launch s4 instances.
> - It still requires deploying node managers on each box which means it will
> be useful if one is running more than one s4 process on a node.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message