crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gabriel Reid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-619) Run on HBase 2
Date Mon, 22 May 2017 15:19:04 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019694#comment-16019694
] 

Gabriel Reid commented on CRUNCH-619:
-------------------------------------

[~pairg] I'm really sorry for taking forever to get back to you on this.

I looked at the changes that the the hbase2_v5 patch introduces around the node affinity stuff
added in CRUNCH-644. It looks like it will still work just fine, but I think that the API
for using it becomes pretty awkward because you have to supply both a Table and a RegionLocator
to the {{HFileUtils.writeXXXToHFilesForIncrementalLoad}} methods.

Seeing as Table and RegionLocator are both bound to a single table, and they are both accessible
via Connection (and this change seems to break source compatibility anyhow), I would suggest
changing those methods to take a Connection and TableName, and then internally retrieve the
Table and RegionLocator from the Connection.

If we want to keep source compatibility, we can just keep the existing version of those methods
(without the RegionLocator) and change the underlying implementations to be able to work with
a null RegionLocator (i.e. just disable the node affinity stuff in that case).

> Run on HBase 2
> --------------
>
>                 Key: CRUNCH-619
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-619
>             Project: Crunch
>          Issue Type: Improvement
>    Affects Versions: 1.0.0
>            Reporter: Tom White
>            Assignee: Gergő Pásztor
>             Fix For: 1.0.0
>
>         Attachments: CRUNCH-619-2.patch, CRUNCH-619.patch, CRUNCH-619_v3.patch, CRUNCH-619_v4_hbase1.patch,
CRUNCH-619_v4_hbase2.patch, CRUNCH-619_v5_hbase2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message