giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maja Kabiljo (JIRA)" <>
Subject [jira] [Updated] (GIRAPH-648) Allow IO formats to add parameters to Configuration
Date Sun, 21 Apr 2013 17:37:15 GMT


Maja Kabiljo updated GIRAPH-648:

    Attachment: GIRAPH-648.patch

Unfortunately this is not as straightforward as it sounds, because methods from io formats,
readers/writers/OutputCommitter have JobContext or TaskAttemptContext as an argument, and
in some cases those hold the copy of Configuration, not the original. So I added a way to
track which parameters where added to GiraphConfiguration, and wrapped all io related calls
to append those parameters to JobContext/TaskAttemptContext before passing control to actual
io formats.

Cleaned up HiveGiraphRunner and moved all control to its io formats, I can do similar for
HCatalog in a separate patch.

This will also help us do GIRAPH-639 in a cleaner way, and it will actually be possible to
mix different kind of input formats (hcat, hive, hbase, or whatever).
> Allow IO formats to add parameters to Configuration
> ---------------------------------------------------
>                 Key: GIRAPH-648
>                 URL:
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Maja Kabiljo
>            Assignee: Maja Kabiljo
>         Attachments: GIRAPH-648.patch
> Currently we heavily rely on some runners (HCatGiraphRunner and HiveGiraphRunner) to
prepare Configuration before application starts, and we have no way of using hcat/hive io
without these runners. It would be better and more flexible if io formats would add what's
needed for underlying io to Configuration themselves.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

View raw message