crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-619) Run on HBase 2
Date Fri, 09 Sep 2016 15:14:20 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15477289#comment-15477289
] 

Jonathan Hsieh commented on CRUNCH-619:
---------------------------------------

Hey [~tomwhite], I took a quick look at the patch and though I haven't tried it, most of the
changes should be able to run against hbase 1.0+ and hbase 2.x when it comes out.  The preferred
HBase 1.x  API is changed between the 0.98/0.96 APIs previously used in crunch, but were still
present in HBase 1.x's.  The Hbase 2.x line will remove the older apis, and thus forces all
components to move to the 1.x api.

The caveat is the HFile readers and writers which you mention aren't part of the public HBase
API [1].  So for the HFile writers, I wonder if it would be possible to wrap or extend the
existing public HBase HFileOutputFormat2 [2] so that you don't have to get into the internals.

A few notes: KeyValue is no longer public and may go away in the future,  (there are equivalent
methods in CellUtil)

Do you all use review board?  I could comment/code review more easily there.


[1] http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/io/hfile/HFile.html
[2] http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.html

> Run on HBase 2
> --------------
>
>                 Key: CRUNCH-619
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-619
>             Project: Crunch
>          Issue Type: Improvement
>    Affects Versions: 0.14.0
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: CRUNCH-619.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message