crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Shi (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CRUNCH-340) HCatSource
Date Sat, 22 Feb 2014 16:41:19 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13909430#comment-13909430
] 

Chao Shi edited comment on CRUNCH-340 at 2/22/14 4:39 PM:
----------------------------------------------------------

[~jwills], I tested against hive 0.13 (not released). It should work on hadoop2 too.

According to \[1\], hive 0.13 will be released pretty soon. So I think this should be ready
for commit after several minor issues addressed (listed below):

1) pick one name: crunch-\{hcat, hcatalog, hive\}, \{HCat, HCatalog, Hive\}Source, From\{HCat,
HCatalog, Hive\}
2) add partition support

Please note that that there is a API change in hive 0.13 \[2\], so there needs a minor change
to get patch this compiled.

\[1\] http://mail-archives.apache.org/mod_mbox/hive-user/201402.mbox/%3CC26507DC-3D95-4AE4-A66F-936648677506%40hortonworks.com%3E
\[2\] HIVE-6185


was (Author: stepinto):
[~jwills], I tested against hive 0.13 (not released). It should work on hadoop2 too.

According to [1], hive 0.13 will be released pretty soon. So I think this should be ready
for commit after several minor issues addressed (listed below):

1) pick one name: crunch-{hcat, hcatalog, hive}, {HCat, HCatalog, Hive}Source, From{HCat,
HCatalog, Hive}
2) add partition support

Please note that that there is a API change in hive 0.13 [2], so there needs a minor change
to get patch this compiled.

[1] http://mail-archives.apache.org/mod_mbox/hive-user/201402.mbox/%3CC26507DC-3D95-4AE4-A66F-936648677506%40hortonworks.com%3E
[2] HIVE-6185

> HCatSource
> ----------
>
>                 Key: CRUNCH-340
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-340
>             Project: Crunch
>          Issue Type: New Feature
>            Reporter: Chao Shi
>         Attachments: crunch-340-v2.patch, crunch-340-v3.patch, crunch-340.patch
>
>
> This patch adds HCatSource, which enables crunch pipeline to read from Hive tables. This
is the very first version, leaving a few TODOs in code.
> It adds new dependency from crunch-core to hcatalog (as well as several hive components).
I guess maybe we should create a new subproject (e.g. crunch-hcatalog) rather than add it
into crunch-core.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message