giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eli Reisman (JIRA)" <>
Subject [jira] [Commented] (GIRAPH-503) Refactor platform-independent CLI argument parsing in GiraphRunner into a separate class
Date Fri, 15 Feb 2013 22:33:13 GMT


Eli Reisman commented on GIRAPH-503:

@nitay: thanks for taking a 2nd look, I did check out HiveGiraphRunner and my thoughts are:

1. this patch doesn't alter anything that will affect the hive runner as far as I could see.
GiraphJob is basically left alone. The only thing you might be losing is some of the validation
but that was already moved to classes called from within GiraphRunner a while back, so you're
already missing out on that.

2. I think the approach to merge the two runners will work really well after this patch is
in. The key will be to add the mod you made to HiveGiraphRunner <code>addMoreOptions()</code>
and <code>processMoreArguments</code> to the base GiraphRunner, then make HiveGiraphRunner
just inherit from GiraphRunner, use those two calls to populate Hive-specific stuff into your
job, and the "new" HiveGiraphRunner will be a very short class.

3. Other approaches will be very possible too, but this inheritance idea would be nice for
HiveGR because it gets the validation checks on the conf back for free, and by getting the
job activation from its GR base class, it will get to run on YARN profile as well. I think
classes that inherit from this new base GR will need to think about job setup in terms of
altering their Configuration and let GiraphJob or GiraphYarnClient (my YarnGiraphJob class)
populate their own job settings from what they find in the parsed conf.

Anyway, this should put us in a good place for those refactors, I can even do the HiveGR one
at some point coming up here, it shouldn't be too hard after this.

> Refactor platform-independent CLI argument parsing in GiraphRunner into a separate class
> ----------------------------------------------------------------------------------------
>                 Key: GIRAPH-503
>                 URL:
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Eli Reisman
>            Assignee: Eli Reisman
>            Priority: Minor
>         Attachments: GIRAPH-503-1.patch, GIRAPH-503-2.patch, GIRAPH-503-3.patch, GIRAPH-503-4.patch,
GIRAPH-503-5.patch, GIRAPH-503-6.patch
> In order to run on non Hadoop MR platforms, we will need to populate the GiraphConfiguration
for our job in a platform-independent way so that all config options are available to whatever
driver class initiates the Giraph job (not just GiraphRunner/GiraphJob.) This also serves
to clean up GiraphRunner in general.
> Passes 'mvn clean install'
> Review Board URL:

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message