hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "roncenzhao (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-14797) reducer number estimating may lead to data skew
Date Tue, 20 Sep 2016 08:37:20 GMT
roncenzhao created HIVE-14797:
---------------------------------

             Summary: reducer number estimating may lead to data skew
                 Key: HIVE-14797
                 URL: https://issues.apache.org/jira/browse/HIVE-14797
             Project: Hive
          Issue Type: Improvement
          Components: Query Processor
            Reporter: roncenzhao
            Assignee: roncenzhao


HiveKey's hash code is generated by multipling by 31 key by key which is implemented in method
`ObjectInspectorUtils.getBucketHashCode()`:
for (int i = 0; i < bucketFields.length; i++) {
      int fieldHash = ObjectInspectorUtils.hashCode(bucketFields[i], bucketFieldInspectors[i]);
      hashCode = 31 * hashCode + fieldHash;
    }

The follow example will lead to data skew:

I hava two table called tbl1 and tbl2 and they have the same column: a int, b string. The
values of column 'a' in both two tables are not skew, but values of column 'b' in both two
tables are skew.

When my sql is "select * from tbl1 join tbl2 on tbl1.a=tbl2.a and tbl1.b=tbl2.b" and the estimated
reducer number is 31, it will lead to data skew.

As we know, the HiveKey's hash code is generated by `hash(a)*31 + hash(b)`. When reducer number
is 31 the reducer No. of each row is `hash(b)%31`. In the result, the job will be skew.





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message