Mailing-List: contact issues-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Wed, 21 Sep 2016 05:17:20 +0000 (UTC)
From: "Rui Li (JIRA)" <jira@apache.org>
To: issues@hive.apache.org
Message-ID: <JIRA.13006144.1474360584000.623389.1474435040549@Atlassian.JIRA>
In-Reply-To: <JIRA.13006144.1474360584000@Atlassian.JIRA>
References: <JIRA.13006144.1474360584000@Atlassian.JIRA> <JIRA.13006144.1474360584575@arcas>
Subject: [jira] [Comment Edited] (HIVE-14797) reducer number estimating may
 lead to data skew
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Wed, 21 Sep 2016 05:17:22 -0000


    [ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508789#comment-15508789 ] 

Rui Li edited comment on HIVE-14797 at 9/21/16 5:16 AM:
--------------------------------------------------------

Hmm random prime won't work because we need to make sure same rows always have same hash code. I can think of another way:
{code}
1. If we have only one field, we can just return the field's hash code.
2. If we have multiple fields, we can compute hash code as: P1*hash(F1)+...+Pn*hash(Fn). Where hash(Fn) is the hash code of the nth field, and {P1,...,Pn} is a deterministic series of prime numbers, e.g. {17,19,...}. Seems BigInteger::nextProbablePrime() can help generate the series.
{code}


was (Author: lirui):
Hmm random prime won't work because we need to make sure same rows always have same hash code. I can think of another way:
{code}
1. If we have only one field, we can just return the field's hash code.
2. If we have multiple fields, we can compute hash code as: P1*hash(F1)+...+Pn*hash(Fn). Where hash(Fn) is the hash code of the nth field, and {P1,...,Pn} is a deterministic series of prime numbers, e.g. {17,19,...}. Seems {{BigInteger::nextProbablePrime()}} can help generate the series.
{code}

> reducer number estimating may lead to data skew
> -----------------------------------------------
>
>                 Key: HIVE-14797
>                 URL: https://issues.apache.org/jira/browse/HIVE-14797
>             Project: Hive
>          Issue Type: Improvement
>          Components: Query Processor
>            Reporter: roncenzhao
>            Assignee: roncenzhao
>         Attachments: HIVE-14797.patch
>
>
> HiveKey's hash code is generated by multipling by 31 key by key which is implemented in method `ObjectInspectorUtils.getBucketHashCode()`:
> for (int i = 0; i < bucketFields.length; i++) {
>       int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i], bucketFieldInspectors[i]);
>       hashCode = 31 * hashCode + fieldHash;
>     }
> The follow example will lead to data skew:
> I hava two table called tbl1 and tbl2 and they have the same column: a int, b string. The values of column 'a' in both two tables are not skew, but values of column 'b' in both two tables are skew.
> When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and tbl1.b=tbl2.b" and the estimated reducer number is 31, it will lead to data skew.
> As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`. When reducer number is 31 the reducer No. of each row is `hash(b)%31`. In the result, the job will be skew.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)