hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-7232) ReduceSink is emitting NULL keys due to failed keyEval
Date Sat, 14 Jun 2014 07:51:01 GMT
Gopal V created HIVE-7232:
-----------------------------

             Summary: ReduceSink is emitting NULL keys due to failed keyEval
                 Key: HIVE-7232
                 URL: https://issues.apache.org/jira/browse/HIVE-7232
             Project: Hive
          Issue Type: Bug
          Components: Query Processor
    Affects Versions: 0.14.0
            Reporter: Gopal V


After HIVE-4867 has been merged in, some queries have exhibited a very weird skew towards
NULL keys emitted from the ReduceSinkOperator.

Added extra logging to print expr.column() in ExprNodeColumnEvaluator & in reduce sink.

{code}
2014-06-14 00:37:19,186 INFO [TezChild] org.apache.hadoop.hive.ql.exec.ReduceSinkOperator:
numDistributionKeys = 1 {null --> ExprNodeColumnEvaluator(_col10)}
key_row={"reducesinkkey0":442}
{code}

{code}
      HiveKey firstKey = toHiveKey(cachedKeys[0], tag, null);
      int distKeyLength = firstKey.getDistKeyLength();
      if(distKeyLength <= 1) {
        StringBuffer x1 = new StringBuffer();
        x1.append("numDistributionKeys = "+ numDistributionKeys + "\n");
        for (int i = 0; i < numDistributionKeys; i++) {
            x1.append(cachedKeys[0][i] + " --> " + keyEval[i] + "\n");
        }
        x1.append("key_row="+ SerDeUtils.getJSONString(row, keyObjectInspector));
        LOG.info("GOPAL: " + x1.toString());
      }
{code}

The query is tpc-h query5, with extra NULL checks just to be sure.

{code}
ELECT n_name,
       sum(l_extendedprice * (1 - l_discount)) AS revenue
FROM customer,
     orders,
     lineitem,
     supplier,
     nation,
     region
WHERE c_custkey = o_custkey
  AND l_orderkey = o_orderkey
  AND l_suppkey = s_suppkey
  AND c_nationkey = s_nationkey
  AND s_nationkey = n_nationkey
  AND n_regionkey = r_regionkey
  AND r_name = 'ASIA'
  AND o_orderdate >= '1994-01-01'
  AND o_orderdate < '1995-01-01'
  and l_orderkey is not null
  and c_custkey is not null
  and l_suppkey is not null
  and c_nationkey is not null
  and s_nationkey is not null
  and n_regionkey is not null
GROUP BY n_name
ORDER BY revenue DESC;
{code}

The reducer which has the issue has the following plan

{code}
Reducer 3
            Reduce Operator Tree:
              Join Operator
                condition map:
                     Inner Join 0 to 1
                condition expressions:
                  0 {KEY.reducesinkkey0} {VALUE._col2}
                  1 {VALUE._col0} {KEY.reducesinkkey0} {VALUE._col3}
                outputColumnNames: _col0, _col3, _col10, _col11, _col14
                Statistics: Num rows: 183333344 Data size: 95229140992 Basic stats: COMPLETE
Column stats: NONE
                Reduce Output Operator
                  key expressions: _col10 (type: int)
                  sort order: +
                  Map-reduce partition columns: _col10 (type: int)
                  Statistics: Num rows: 183333344 Data size: 95229140992 Basic stats: COMPLETE
Column stats: NONE
                  value expressions: _col0 (type: int), _col3 (type: int), _col11 (type: int),
_col14 (type: string)
{code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message