hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yin Huai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-4502) NPE - subquery smb joins fails
Date Sun, 21 Jul 2013 18:08:48 GMT

    [ https://issues.apache.org/jira/browse/HIVE-4502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13714762#comment-13714762
] 

Yin Huai commented on HIVE-4502:
--------------------------------

The failed query in correlationoptimizer3.q is ...
{code:sql}
set hive.optimize.correlation=true;
set hive.auto.convert.join=true;
set hive.optimize.mapjoin.mapreduce=true;

SELECT d.key AS key, d.cnt AS cnt, b.value as value
FROM (SELECT x.key as key, x.value as value FROM src1 x JOIN src y ON (x.key = y.key)) b
JOIN (SELECT x.key as key, count(1) AS cnt FROM src1 x JOIN src y ON (x.key = y.key) group
by x.key) d
ON (b.key = d.key);
{\code}
and here is the stack trace ...
{code}
2013-07-21 10:14:04,440 INFO  lazybinary.LazyBinaryStruct (LazyBinaryStruct.java:parse(172))
- Missing fields! Expected 1 fields but only got 0! Ignoring similar problems.
2013-07-21 10:14:04,444 FATAL ExecReducer (ExecReducer.java:reduce(269)) - org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row (tag=0) {"key":{"joinkey0":"128"},"value":{"_col1":""}}
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:258)
	at org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:520)
	at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:421)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:262)
Caused by: java.lang.ArrayIndexOutOfBoundsException
	at java.lang.System.arraycopy(Native Method)
	at org.apache.hadoop.io.Text.set(Text.java:205)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:216)
	at org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:197)
	at org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:61)
	at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
	at org.apache.hadoop.hive.serde2.SerDeUtils.buildJSONString(SerDeUtils.java:371)
	at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:236)
	at org.apache.hadoop.hive.serde2.SerDeUtils.getJSONString(SerDeUtils.java:222)
	at org.apache.hadoop.hive.ql.exec.MuxOperator.processOp(MuxOperator.java:249)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
	at org.apache.hadoop.hive.ql.exec.DemuxOperator.processOp(DemuxOperator.java:245)
	at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:504)
	at org.apache.hadoop.hive.ql.exec.mr.ExecReducer.reduce(ExecReducer.java:249)
	... 3 more
{\code}

At here, DemuxOperator only replaces the tag (the tag seems correct at here) and forward the
row to MuxOperator. Then, MuxOperator will just forward the row to the JoinOperator. Seems
we cannot get the value in this case... Still looking for the root cause... Seems related
to MapJoin because the query with hive.optimize.correlation=true and hive.auto.convert.join=false
is fine.
                
> NPE - subquery smb joins fails
> ------------------------------
>
>                 Key: HIVE-4502
>                 URL: https://issues.apache.org/jira/browse/HIVE-4502
>             Project: Hive
>          Issue Type: Bug
>          Components: Query Processor
>    Affects Versions: 0.11.0
>            Reporter: Vikram Dixit K
>            Assignee: Navis
>         Attachments: HIVE-4502.D10695.1.patch, HIVE-4502.D10695.2.patch, HIVE-4502.D10695.3.patch,
HIVE-4502.D10695.4.patch, HIVE-4502.patch, smb_mapjoin_25.q, smb_mapjoin_25.q
>
>
> Found this issue while running some SMB joins. Attaching test case that causes this error.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message