incubator-s4-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthieu Morel (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (S4-25) Write S4 Application Master to deploy S4 in Yarn
Date Mon, 26 Nov 2012 16:49:00 GMT

    [ https://issues.apache.org/jira/browse/S4-25?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13503888#comment-13503888
] 

Matthieu Morel commented on S4-25:
----------------------------------

Thanks for the comments, I've uploaded a new version of the patch in branch S4-25, commit
830f5df

By default, Application Master and S4 node JVMs are started with a maximum heap size of 80%
the memory for the container. This can be overridden through JVM parameters, when the user
has specific knowledge about the application. We still use Xmx to compute the resource allocation
though, since, AFAIK, we cannot compute the exact memory usage automatically with available
parameters.

If the specified memory exceeds the available memory, Yarn will simply log an error message
and not start the JVMs.
                
> Write S4 Application Master to deploy S4 in Yarn
> ------------------------------------------------
>
>                 Key: S4-25
>                 URL: https://issues.apache.org/jira/browse/S4-25
>             Project: Apache S4
>          Issue Type: New Feature
>            Reporter: J Mohamed Zahoor
>             Fix For: 0.6
>
>         Attachments: S4-ApplicationMaster.diff, S4-Client.diff, S4-Constants.diff, S4-YARN-1.patch
>
>
> On the lines of s4PigWrapper, write a s4 application master to host s4 piper inside Hadoop
Yarn. This could be useful not only for reading data stored in hadoop ( to build or train
a model)... But we could make use of the resource manager to deploy s4 instances in remote
machine and monitor them. In short, we could make use of most of the resource management ,
scheduling and other good stuff in Yarn.
> - Yarn is useful to deploy and launch s4 instances.
> - It still requires deploying node managers on each box which means it will
> be useful if one is running more than one s4 process on a node.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message