Return-Path: 
 <dev-return-95727-apmail-hive-dev-archive=hive.apache.org@hive.apache.org>
X-Original-To: apmail-hive-dev-archive@www.apache.org
Delivered-To: apmail-hive-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 539A51128B
	for <apmail-hive-dev-archive@www.apache.org>;
 Mon,  4 Aug 2014 15:24:13 +0000 (UTC)
Received: (qmail 40598 invoked by uid 500); 4 Aug 2014 15:24:12 -0000
Delivered-To: apmail-hive-dev-archive@hive.apache.org
Received: (qmail 40531 invoked by uid 500); 4 Aug 2014 15:24:12 -0000
Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@hive.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@hive.apache.org>
List-Post: <mailto:dev@hive.apache.org>
List-Id: <dev.hive.apache.org>
Reply-To: dev@hive.apache.org
Delivered-To: mailing list dev@hive.apache.org
Received: (qmail 40506 invoked by uid 500); 4 Aug 2014 15:24:12 -0000
Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org
Received: (qmail 40503 invoked by uid 99); 4 Aug 2014 15:24:12 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 04 Aug 2014 15:24:12 +0000
Date: Mon, 4 Aug 2014 15:24:12 +0000 (UTC)
From: "Brock Noland (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12730090.1406492244267.6742.1407165852476@arcas>
In-Reply-To: <JIRA.12730090.1406492244267@arcas>
References: <JIRA.12730090.1406492244267@arcas>
Subject: [jira] [Commented] (HIVE-7526) Research to use groupby
 transformation to replace Hive existing partitionByKey and SparkCollector
 combination
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HIVE-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084766#comment-14084766 ] 

Brock Noland commented on HIVE-7526:
------------------------------------

Hi [~lirui],

I am might be misunderstanding...but it looks like it's calculated automatically in HIVE-7567.

> Research to use groupby transformation to replace Hive existing partitionByKey and SparkCollector combination
> -------------------------------------------------------------------------------------------------------------
>
>                 Key: HIVE-7526
>                 URL: https://issues.apache.org/jira/browse/HIVE-7526
>             Project: Hive
>          Issue Type: Task
>          Components: Spark
>            Reporter: Xuefu Zhang
>            Assignee: Chao
>             Fix For: spark-branch
>
>         Attachments: HIVE-7526.2.patch, HIVE-7526.3.patch, HIVE-7526.4-spark.patch, HIVE-7526.5-spark.patch, HIVE-7526.patch
>
>
> Currently SparkClient shuffles data by calling paritionByKey(). This transformation outputs <key, value> tuples. However, Hive's ExecMapper expects <key, iterator<value>> tuples, and Spark's groupByKey() seems outputing this directly. Thus, using groupByKey, we may be able to avoid its own key clustering mechanism (in HiveReduceFunction). This research is to have a try.



--
This message was sent by Atlassian JIRA
(v6.2#6252)