giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (GIRAPH-574) Move Giraph Master node functionality to AppMaster or launch directly from AppMaster in YARN profile
Date Sun, 17 Mar 2013 00:50:13 GMT

    [ https://issues.apache.org/jira/browse/GIRAPH-574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13604468#comment-13604468
] 

Eli Reisman commented on GIRAPH-574:
------------------------------------

Think aggregators, master compute, other things? also...

                
> Move Giraph Master node functionality to AppMaster or launch directly from AppMaster
in YARN profile
> ----------------------------------------------------------------------------------------------------
>
>                 Key: GIRAPH-574
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-574
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Eli Reisman
>            Priority: Minor
>
> As folks read the Giraph on YARN code it is inevitable it will occur to someone "Well,
if the job fails when the ApplicationMaster fails, could we move some or all of our Master
task functions there and just call it master?"
> Yes. In two ways.
> One, we launch a dedicated master process marked as such with setup responsibilities,
and we assess from the app master how the launch went. We keep launching "masters" until one
takes. Then, we launch the workers.
> Another is to simply run MasterThread and associated stuff from the App Master directly,
and when we know its up and running properly, only then does app master launch the workers.
> The YARN app master can be rebooted is designed to be a place for fault-tolerant "master
node" stuff to happen. However, I think a larger purpose is to act as a meta-master for launching
a DAG of jobs within the run of a single app master lifecycle. Or the app master cant act
as any of these things, or something else I haven't thought of. The architecture is fairly
malleable.
> This is not a requirement for us, and maybe not a good idea at all. This is just a placeholder
JIRA to discuss and collect ideas since as I said above someone is going to bring it up ;)
> Thank you for reading.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message