drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aman Sinha (JIRA)" <j...@apache.org>
Subject [jira] [Created] (DRILL-2803) Severe skew due to null values in columns even when other columns are non-null
Date Thu, 16 Apr 2015 02:45:58 GMT
Aman Sinha created DRILL-2803:
---------------------------------

             Summary: Severe skew due to null values in columns even when other columns are
non-null
                 Key: DRILL-2803
                 URL: https://issues.apache.org/jira/browse/DRILL-2803
             Project: Apache Drill
          Issue Type: Bug
          Components: Functions - Drill
    Affects Versions: 0.8.0
            Reporter: Aman Sinha
            Assignee: Jacques Nadeau


If you have 2 columns that are hashed (either for distribution or for hash based operators)
and one of those columns has lots of null values, it can result in substantial skew even if
the other column has non-null values. 

In the following query the combined hash value of 2 columns is 0 even when 1 column is non-null.
  The reason is that if the starting value is null (for cr_reason_sk all values are null in
the above query), it does not matter what seed is passed in.   The hash function treats the
second parameter as a seed and not as a combiner, so it gets ignored. 

{code}
select cr_call_center_sk, cr_reason_sk, hash64(cr_reason_sk, hash64(cr_call_center_sk)) as
hash_value from catalog_returns  where cr_reason_sk is null and cr_call_center_sk is not null
limit 10;
+-------------------+--------------+------------+
| cr_call_center_sk | cr_reason_sk | hash_value |
+-------------------+--------------+------------+
| 1                 | null         | 0          |
| 1                 | null         | 0          |
| 4                 | null         | 0          |
| 1                 | null         | 0          |
| 4                 | null         | 0          |
| 2                 | null         | 0          |
| 2                 | null         | 0          |
| 2                 | null         | 0          |
| 2                 | null         | 0          |
| 2                 | null         | 0          |
+-------------------+--------------+------------+
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message