giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <>
Subject [jira] [Updated] (GIRAPH-479) Factor creation of job-local ZooKeeper instances out of ZKManager to enable YARN instantiation, etc.
Date Thu, 21 Feb 2013 00:45:12 GMT


Eli Reisman updated GIRAPH-479:

    Attachment: GIRAPH-479-4.patch

Just a rebase. This is a nice refactor but may not be needed to make YARN work, so its low
priority until more of that work is done. This does work, and passes mvm verify if anyone
does see a compelling reason to move on it sooner.

It is very possible (even easy) when the YARN impl is up and running that we can just have
our ApplicationMaster launch an instance of ZK if the giraph.zkList conf field is empty, and
then populate that field with the host and port of our new ZK instance to "mimic" the local
instance we use on MRv1. When the workers are instantiated, they will already have the correct
ZK address in their conf and won't even know if its an app-local or standalone instance! Whenever
the ZkManager gets fired up, it sees zkList is populated and just goes back to sleep, allowing
YARN to handle this part of the ZK function without changing the MRv1 ZK code at all.

We could even delay the rest of the launches for a bit to guarantee its up and running before
we start our workers. So there's a number of things brewing around this right now that might
make it nice but not needed. But I digress...

> Factor creation of job-local ZooKeeper instances out of ZKManager to enable YARN instantiation,
> ----------------------------------------------------------------------------------------------------
>                 Key: GIRAPH-479
>                 URL:
>             Project: Giraph
>          Issue Type: Improvement
>          Components: zookeeper
>    Affects Versions: 0.2.0
>            Reporter: Eli Reisman
>            Assignee: Eli Reisman
>            Priority: Minor
>         Attachments: GIRAPH-479-1.patch, GIRAPH-479-2.patch, GIRAPH-479-3.patch, GIRAPH-479-4.patch
> When the user does not specify a ZooKeeper quorum to run Giraph on, the ZooKeeperManager
creates one. This factors out the boilerplate that handles this additional process and its
lifecycle so that various pluggable versions of this process instantiation can be implemented
and fed to the ZKManager via a factory depending on the nature of the underlying cluster.
> By implementing the ZooKeeperProcessBuilder interface, one will now be able to implement
the creation/mangement of a new, job-local ZK instance using any underlying resourcing mechanism
one chooses (including, I hope, YARN.)
> Passes mvn verify, etc.
> Review board link:

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message