Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Tue, 9 Sep 2014 16:09:28 +0000 (UTC)
From: "Xuefu Zhang (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12739743.1410056692000.28432.1410278968944@Atlassian.JIRA>
In-Reply-To: <JIRA.12739743.1410056692000@Atlassian.JIRA>
References: <JIRA.12739743.1410056692000@Atlassian.JIRA>
 <JIRA.12739743.1410056692287@arcas>
Subject: [jira] [Commented] (HIVE-8017) Use HiveKey instead of BytesWritable
 as key type of the pair RDD [Spark Branch]
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HIVE-8017?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14127=
157#comment-14127157 ]=20

Xuefu Zhang commented on HIVE-8017:
-----------------------------------

Hi Rui,

{quote}
Have you tried =E2=80=93 SORT_BEFORE_DIFF, which is to sort the query resul=
t, which is different from SORT_BEFORE_DIFF. It's supposed to make query re=
sult sorted w/o an order by.
{quote}

This comment indeeded made no sense. Sorry for the confusion, I meant "-- S=
ORT_QUERY_RESULTS" for the first "-- SORT_BEFORE_DIFF". So my comment shoul=
d really go like this:

{bold}
Have you tried =E2=80=93 SORT_QUERY_RESULTS, which is to sort the query res=
ult, which is different from SORT_BEFORE_DIFF. It's supposed to make query =
result sorted w/o an order by.
{bold}

 SORT_BEFORE_DIFF sorts the output files before making diff. It's less reli=
able, and sometimes the diff doesn't tell what's wrong. Thus, I think we sh=
ould prefer SORT_QUERY_RESULTS when query output can diff in order.=20

Neither do I know why SORT_BEFORE_DIFF has to come after set commands in yo=
ur case, but it seems the usage of it in the .q files is not consistent on =
that front.


> Use HiveKey instead of BytesWritable as key type of the pair RDD [Spark B=
ranch]
> -------------------------------------------------------------------------=
------
>
>                 Key: HIVE-8017
>                 URL: https://issues.apache.org/jira/browse/HIVE-8017
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Rui Li
>            Assignee: Rui Li
>         Attachments: HIVE-8017-spark.patch, HIVE-8017.2-spark.patch
>
>
> HiveKey should be used as the key type because it holds the hash code for=
 partitioning. While BytesWritable serves partitioning well for simple case=
s, we have to use {{HiveKey.hashCode}} for more complicated ones, e.g. join=
, bucketed table, etc.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)