hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Taklon Stephen Wu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-14121) Error occurs when it use the Tez, MapJoin and Vectorization.
Date Wed, 29 Jun 2016 21:27:10 GMT

    [ https://issues.apache.org/jira/browse/HIVE-14121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15355817#comment-15355817
] 

Taklon Stephen Wu commented on HIVE-14121:
------------------------------------------

I got a similar error when running some queries (e.g. query17) of a large (10TB) TPC-DS dataset
in parquet format on HDFS without setting hive.auto.convert.join.noconditionaltask.size, the
root line is ReduceRecordSource.pushRecord instead of MapRecordSource.processRow. after setting
hive.auto.convert.join.noconditionaltask.size to 300MB, some of the failed queries passed,
but few of them (e.g. query88) still failed with the same Null Pointer exception.

Vertex failed, vertexName=Reducer 3, vertexId=vertex_1466060732175_0149_1_09, diagnostics=[Task
failed, taskId=task_1466060732175_0149_1_09_000174, diagnostics=[TaskAttempt 0 failed, info=[Error:
Error while running task ( failure ) : attempt_1466060732175_0149_1_09_000174_0:java.lang.RuntimeException:
java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
Error while processing row (tag=0) {"key":{"reducesinkkey0":1432165,"reducesinkkey1":282176},"value":{"_col0":282176,"_col2":214,"_col4":21,"_col5":2451624,"_col7":19,"_col8":2451676}}
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:198)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:355)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)
at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
Runtime Error while processing row (tag=0) {"key":{"reducesinkkey0":1432165,"reducesinkkey1":282176},"value":{"_col0":282176,"_col2":214,"_col4":21,"_col5":2451624,"_col7":19,"_col8":2451676}}
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:288)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:263)
at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
... 14 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing
row (tag=0) {"key":{"reducesinkkey0":1432165,"reducesinkkey1":282176},"value":{"_col0":282176,"_col2":214,"_col4":21,"_col5":2451624,"_col7":19,"_col8":2451676}}
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:356)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:278)
... 16 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception from MapJoinOperator
: Unexpected exception from MapJoinOperator : null
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.internalForward(CommonJoinOperator.java:647)
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genAllOneUniqueJoinObject(CommonJoinOperator.java:679)
at org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:757)
at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinObject(CommonMergeJoinOperator.java:316)
at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:279)
at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:272)
at org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:258)
at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:347)
... 17 more
Caused by: java.lang.NullPointerException
at org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer$ReusableRowContainer.setFromOutput(HybridHashTableContainer.java:922)
at org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer$GetAdaptor.setFromRow(HybridHashTableContainer.java:802)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.setMapJoinKey(MapJoinOperator.java:339)
at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:390)

> Error occurs when it use the Tez, MapJoin and Vectorization.
> ------------------------------------------------------------
>
>                 Key: HIVE-14121
>                 URL: https://issues.apache.org/jira/browse/HIVE-14121
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 2.1.0
>            Reporter: Ryu Kobayashi
>            Priority: Critical
>
> This issue is 2.1.0 occurs.
> {code}
> 2016-06-29 14:29:26,884 [ERROR] [TezChild] |tez.MapRecordProcessor|: Hit error while
closing operators - failing tree
> 2016-06-29 14:29:26,884 [ERROR] [TezChild] |tez.TezProcessor|: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row
{"xxx":"yyy"}
> 	at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> 	at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> 	at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:360)
> 	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:172)
> 	at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:160)
> 	at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:355)
> 	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:72)
> 	at org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:60)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:415)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> 	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:60)
> 	at org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:36)
> 	at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> 	at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {"xxx":"yyy"}
> 	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:499)
> 	at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> 	... 17 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unexpected exception from
MapJoinOperator : null
> 	at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:454)
> 	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
> 	at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95)
> 	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
> 	at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:126)
> 	at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:878)
> 	at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> 	at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:149)
> 	at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:489)
> 	... 18 more
> Caused by: java.lang.NullPointerException
> 	at org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer$ReusableRowContainer.setFromOutput(HybridHashTableContainer.java:922)
> 	at org.apache.hadoop.hive.ql.exec.persistence.HybridHashTableContainer$GetAdaptor.setFromRow(HybridHashTableContainer.java:802)
> 	at org.apache.hadoop.hive.ql.exec.MapJoinOperator.setMapJoinKey(MapJoinOperator.java:339)
> 	at org.apache.hadoop.hive.ql.exec.MapJoinOperator.process(MapJoinOperator.java:390)
> 	... 26 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message