Return-Path: 
 <dev-return-3869-apmail-crunch-dev-archive=crunch.apache.org@crunch.apache.org>
X-Original-To: apmail-crunch-dev-archive@www.apache.org
Delivered-To: apmail-crunch-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 249D5106F4
	for <apmail-crunch-dev-archive@www.apache.org>;
 Mon, 10 Feb 2014 16:38:25 +0000 (UTC)
Received: (qmail 97856 invoked by uid 500); 10 Feb 2014 16:38:21 -0000
Delivered-To: apmail-crunch-dev-archive@crunch.apache.org
Received: (qmail 97814 invoked by uid 500); 10 Feb 2014 16:38:20 -0000
Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@crunch.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@crunch.apache.org>
List-Post: <mailto:dev@crunch.apache.org>
List-Id: <dev.crunch.apache.org>
Reply-To: dev@crunch.apache.org
Delivered-To: mailing list dev@crunch.apache.org
Received: (qmail 97769 invoked by uid 500); 10 Feb 2014 16:38:19 -0000
Delivered-To: apmail-incubator-crunch-dev@incubator.apache.org
Received: (qmail 97760 invoked by uid 99); 10 Feb 2014 16:38:19 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 10 Feb 2014 16:38:19 +0000
Date: Mon, 10 Feb 2014 16:38:19 +0000 (UTC)
From: "Chao Shi (JIRA)" <jira@apache.org>
To: crunch-dev@incubator.apache.org
Message-ID: <JIRA.12694162.1391963194543.17868.1392050299383@arcas>
In-Reply-To: <JIRA.12694162.1391963194543@arcas>
References: <JIRA.12694162.1391963194543@arcas>
Subject: [jira] [Updated] (CRUNCH-340) HCatSource
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


     [ https://issues.apache.org/jira/browse/CRUNCH-340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chao Shi updated CRUNCH-340:
----------------------------

    Attachment: crunch-340-v2.patch

Here is v2 patch.

ChangeLog:
- add equals() and hashCode()
- temporarliy hack WritableDeepCopier to get HCatRecord work (*)
- more test cases on multiple input sources

(*) The problem is HCatRecord is an interface, DefaultHCatRecord and LazyHCatRecord are the implementations, which are both writable. This is a temporary hack. I think a better approach is to specify a custom DeepCopier to WritableType of HCatRecord.

Of course, the fallback way is to use HCatSource<DefaultHCatRecord> everywehere, but HCatSource<HCatRecord> definately looks better. I'm further thinking about HCatSource<HCatRecordable>, which looks like a even cleaner interface.

> HCatSource
> ----------
>
>                 Key: CRUNCH-340
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-340
>             Project: Crunch
>          Issue Type: New Feature
>            Reporter: Chao Shi
>         Attachments: crunch-340-v2.patch, crunch-340.patch
>
>
> This patch adds HCatSource, which enables crunch pipeline to read from Hive tables. This is the very first version, leaving a few TODOs in code.
> It adds new dependency from crunch-core to hcatalog (as well as several hive components). I guess maybe we should create a new subproject (e.g. crunch-hcatalog) rather than add it into crunch-core.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)