hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt McCline (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8474) Vectorized reads of transactional tables fail when not all columns are selected
Date Mon, 20 Oct 2014 00:28:33 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14176496#comment-14176496
] 

Matt McCline commented on HIVE-8474:
------------------------------------


Some background on vectorization.

There are are "shadow" VectorizationContext data structures that track which columns of vectorized
row batches for used by each vectorized operators.

In row-by-row mode an operator can easily form a new row Object Array to correspond to the
outputObjInspector.

However, in Vectorization we mask or project away columns in a VectorizedRowBatch (e.g. VectorFilterOperator)
so the same batch can travel down the operators without being copied.  Or, in the case of
computing new columns, VectorSelectOperator will compute new scratch columns.

So, the VectorizationContext starts as all the table columns for Map or the keys and values
for Reduce and then as we go down the operators new VectorizationContext objects are cloned
and their column and scratch column maps are modified.

So, some operators do not use inputObjInspectors or outputObjInspector.  Others, do use them
when the vector operator unpacks batches into rows to call an row mode operator.

> Vectorized reads of transactional tables fail when not all columns are selected
> -------------------------------------------------------------------------------
>
>                 Key: HIVE-8474
>                 URL: https://issues.apache.org/jira/browse/HIVE-8474
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions, Vectorization
>    Affects Versions: 0.14.0
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>            Priority: Critical
>             Fix For: 0.14.0
>
>         Attachments: HIVE-8474.patch
>
>
> {code}
> create table concur_orc_tab(name varchar(50), age int, gpa decimal(3, 2)) clustered by
(age) into 2 buckets stored as orc TBLPROPERTIES ('transactional'='true');
> select name, age from concur_orc_tab order by name;
> {code}
> results in
> {code}
> Diagnostic Messages for this Task:
> Error: java.io.IOException: java.lang.NullPointerException
>         at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
>         at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
>         at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:352)
>         at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:79)
>         at org.apache.hadoop.hive.ql.io.HiveRecordReader.doNext(HiveRecordReader.java:33)
>         at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:115)
>         at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
>         at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.setNullColIsNullValue(VectorizedBatchUtil.java:63)
>         at org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.addRowToBatchFrom(VectorizedBatchUtil.java:443)
>         at org.apache.hadoop.hive.ql.exec.vector.VectorizedBatchUtil.addRowToBatch(VectorizedBatchUtil.java:214)
>         at org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowReader.next(VectorizedOrcAcidRowReader.java:95)
>         at org.apache.hadoop.hive.ql.io.orc.VectorizedOrcAcidRowReader.next(VectorizedOrcAcidRowReader.java:43)
>         at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:347)
>         ... 13 more
> {code}
> The issue is that the object inspector passed to VectorizedOrcAcidRowReader has all of
the columns in the file rather than only the projected columns.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message