hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14797) reducer number estimating may lead to data skew
Date Wed, 21 Sep 2016 16:03:20 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15510378#comment-15510378
] 

Xuefu Zhang commented on HIVE-14797:
------------------------------------

The new change seems good. Minor nit: can we change the implementation of getBucketHashCode()
to call the new method with a seed of 31. This is to save some code duplication.

> reducer number estimating may lead to data skew
> -----------------------------------------------
>
>                 Key: HIVE-14797
>                 URL: https://issues.apache.org/jira/browse/HIVE-14797
>             Project: Hive
>          Issue Type: Improvement
>          Components: Query Processor
>            Reporter: roncenzhao
>            Assignee: roncenzhao
>         Attachments: HIVE-14797.2.patch, HIVE-14797.patch
>
>
> HiveKey's hash code is generated by multipling by 31 key by key which is implemented
in method `ObjectInspectorUtils.getBucketHashCode()`:
> for (int i = 0; i < bucketFields.length; i++) {
>       int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i], bucketFieldInspectors[i]);
>       hashCode = 31 * hashCode + fieldHash;
>     }
> The follow example will lead to data skew:
> I hava two table called tbl1 and tbl2 and they have the same column: a int, b string.
The values of column 'a' in both two tables are not skew, but values of column 'b' in both
two tables are skew.
> When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and tbl1.b=tbl2.b" and
the estimated reducer number is 31, it will lead to data skew.
> As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`. When reducer
number is 31 the reducer No. of each row is `hash(b)%31`. In the result, the job will be skew.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message