giraph-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Maja Kabiljo (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (GIRAPH-648) Allow IO formats to add parameters to Configuration
Date Sun, 21 Apr 2013 17:37:15 GMT

     [ https://issues.apache.org/jira/browse/GIRAPH-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Maja Kabiljo updated GIRAPH-648:
--------------------------------

    Attachment: GIRAPH-648.patch

Unfortunately this is not as straightforward as it sounds, because methods from io formats,
readers/writers/OutputCommitter have JobContext or TaskAttemptContext as an argument, and
in some cases those hold the copy of Configuration, not the original. So I added a way to
track which parameters where added to GiraphConfiguration, and wrapped all io related calls
to append those parameters to JobContext/TaskAttemptContext before passing control to actual
io formats.

Cleaned up HiveGiraphRunner and moved all control to its io formats, I can do similar for
HCatalog in a separate patch.

This will also help us do GIRAPH-639 in a cleaner way, and it will actually be possible to
mix different kind of input formats (hcat, hive, hbase, or whatever).
                
> Allow IO formats to add parameters to Configuration
> ---------------------------------------------------
>
>                 Key: GIRAPH-648
>                 URL: https://issues.apache.org/jira/browse/GIRAPH-648
>             Project: Giraph
>          Issue Type: Improvement
>            Reporter: Maja Kabiljo
>            Assignee: Maja Kabiljo
>         Attachments: GIRAPH-648.patch
>
>
> Currently we heavily rely on some runners (HCatGiraphRunner and HiveGiraphRunner) to
prepare Configuration before application starts, and we have no way of using hcat/hive io
without these runners. It would be better and more flexible if io formats would add what's
needed for underlying io to Configuration themselves.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message