Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Sun, 15 Jun 2014 22:13:02 +0000 (UTC)
From: "Gopal V (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12721236.1402732251809.135030.1402870382186@arcas>
In-Reply-To: <JIRA.12721236.1402732251809@arcas>
References: <JIRA.12721236.1402732251809@arcas>
Subject: [jira] [Commented] (HIVE-7232) ReduceSink is emitting NULL keys due
 to failed keyEval
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HIVE-7232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14032033#comment-14032033 ] 

Gopal V commented on HIVE-7232:
-------------------------------

[~ashutoshc]: Incorrect results as well.

Ran the same query with Tez & MR, got different results.

MR doesn't hit the same scenario becuase of the empty Map task, which doesn't have any input columns named reducesinkkey0.

Tez seems to hit a corner case where there are 2 shuffle joins one after the other - there is an input col named KEY.reducesinkkey0 and an output col named reducesinkkey0, which have no relation to each other.

{code}
$ diff -y -W 72  results/q5.tez.txt results/q5.mr.txt 
CHINA   985314.0848                |    VIETNAM 1.897236998313891E10
INDIA   819113.4418000001          |    CHINA   1.894405687452681E10
VIETNAM 637407.2255                |    INDONESIA       1.89306456994551
JAPAN   523754.9791                |    JAPAN   1.892184676125508E10
INDONESIA       517900.1924        |    INDIA   1.886882412417209E10
{code}

> ReduceSink is emitting NULL keys due to failed keyEval
> ------------------------------------------------------
>
>                 Key: HIVE-7232
>                 URL: https://issues.apache.org/jira/browse/HIVE-7232
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.14.0
>            Reporter: Gopal V
>
> After HIVE-4867 has been merged in, some queries have exhibited a very weird skew towards NULL keys emitted from the ReduceSinkOperator.
> Added extra logging to print expr.column() in ExprNodeColumnEvaluator & in reduce sink.
> {code}
> 2014-06-14 00:37:19,186 INFO [TezChild] org.apache.hadoop.hive.ql.exec.ReduceSinkOperator:
> numDistributionKeys = 1 {null --> ExprNodeColumnEvaluator(_col10)}
> key_row={"reducesinkkey0":442}
> {code}
> {code}
>       HiveKey firstKey = toHiveKey(cachedKeys[0], tag, null);
>       int distKeyLength = firstKey.getDistKeyLength();
>       if(distKeyLength <= 1) {
>         StringBuffer x1 = new StringBuffer();
>         x1.append("numDistributionKeys = "+ numDistributionKeys + "\n");
>         for (int i = 0; i < numDistributionKeys; i++) {
>             x1.append(cachedKeys[0][i] + " --> " + keyEval[i] + "\n");
>         }
>         x1.append("key_row="+ SerDeUtils.getJSONString(row, keyObjectInspector));
>         LOG.info("GOPAL: " + x1.toString());
>       }
> {code}
> The query is tpc-h query5, with extra NULL checks just to be sure.
> {code}
> ELECT n_name,
>        sum(l_extendedprice * (1 - l_discount)) AS revenue
> FROM customer,
>      orders,
>      lineitem,
>      supplier,
>      nation,
>      region
> WHERE c_custkey = o_custkey
>   AND l_orderkey = o_orderkey
>   AND l_suppkey = s_suppkey
>   AND c_nationkey = s_nationkey
>   AND s_nationkey = n_nationkey
>   AND n_regionkey = r_regionkey
>   AND r_name = 'ASIA'
>   AND o_orderdate >= '1994-01-01'
>   AND o_orderdate < '1995-01-01'
>   and l_orderkey is not null
>   and c_custkey is not null
>   and l_suppkey is not null
>   and c_nationkey is not null
>   and s_nationkey is not null
>   and n_regionkey is not null
> GROUP BY n_name
> ORDER BY revenue DESC;
> {code}
> The reducer which has the issue has the following plan
> {code}
> Reducer 3
>             Reduce Operator Tree:
>               Join Operator
>                 condition map:
>                      Inner Join 0 to 1
>                 condition expressions:
>                   0 {KEY.reducesinkkey0} {VALUE._col2}
>                   1 {VALUE._col0} {KEY.reducesinkkey0} {VALUE._col3}
>                 outputColumnNames: _col0, _col3, _col10, _col11, _col14
>                 Statistics: Num rows: 183333344 Data size: 95229140992 Basic stats: COMPLETE Column stats: NONE
>                 Reduce Output Operator
>                   key expressions: _col10 (type: int)
>                   sort order: +
>                   Map-reduce partition columns: _col10 (type: int)
>                   Statistics: Num rows: 183333344 Data size: 95229140992 Basic stats: COMPLETE Column stats: NONE
>                   value expressions: _col0 (type: int), _col3 (type: int), _col11 (type: int), _col14 (type: string)
> {code}


--
This message was sent by Atlassian JIRA
(v6.2#6252)