crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao Shi (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-340) HCatSource
Date Mon, 10 Feb 2014 16:38:19 GMT


Chao Shi updated CRUNCH-340:

    Attachment: crunch-340-v2.patch

Here is v2 patch.

- add equals() and hashCode()
- temporarliy hack WritableDeepCopier to get HCatRecord work (*)
- more test cases on multiple input sources

(*) The problem is HCatRecord is an interface, DefaultHCatRecord and LazyHCatRecord are the
implementations, which are both writable. This is a temporary hack. I think a better approach
is to specify a custom DeepCopier to WritableType of HCatRecord.

Of course, the fallback way is to use HCatSource<DefaultHCatRecord> everywehere, but
HCatSource<HCatRecord> definately looks better. I'm further thinking about HCatSource<HCatRecordable>,
which looks like a even cleaner interface.

> HCatSource
> ----------
>                 Key: CRUNCH-340
>                 URL:
>             Project: Crunch
>          Issue Type: New Feature
>            Reporter: Chao Shi
>         Attachments: crunch-340-v2.patch, crunch-340.patch
> This patch adds HCatSource, which enables crunch pipeline to read from Hive tables. This
is the very first version, leaving a few TODOs in code.
> It adds new dependency from crunch-core to hcatalog (as well as several hive components).
I guess maybe we should create a new subproject (e.g. crunch-hcatalog) rather than add it
into crunch-core.

This message was sent by Atlassian JIRA

View raw message