hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matt McCline (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11981) ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
Date Sat, 24 Oct 2015 20:05:27 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972835#comment-14972835
] 

Matt McCline commented on HIVE-11981:
-------------------------------------

[~prasanth_j] I removed some of the intrusive changes.  Now, no changes to ReaderImpl and
one line change to RecordReaderImpl.  Common code (new class SchemaEvolution) for determining
"schema on read" is called from two places: RecordReaderFactory and OrcRawRecordMerger.  TreeReaderFactory
still has code for determining which columns to null since it seems to be in the best position
to know how to do it instead of RecordReaderImpl...

Latest Hive QA run appears to be successful.

> ORC Schema Evolution Issues (Vectorized, ACID, and Non-Vectorized)
> ------------------------------------------------------------------
>
>                 Key: HIVE-11981
>                 URL: https://issues.apache.org/jira/browse/HIVE-11981
>             Project: Hive
>          Issue Type: Bug
>          Components: Hive, Transactions
>            Reporter: Matt McCline
>            Assignee: Matt McCline
>            Priority: Critical
>         Attachments: HIVE-11981.01.patch, HIVE-11981.02.patch, HIVE-11981.03.patch, HIVE-11981.05.patch,
HIVE-11981.06.patch, HIVE-11981.07.patch, HIVE-11981.08.patch, HIVE-11981.09.patch, HIVE-11981.091.patch,
ORC Schema Evolution Issues.docx
>
>
> High priority issues with schema evolution for the ORC file format.
> Schema evolution here is limited to adding new columns and a few cases of column type-widening
(e.g. int to bigint).
> Renaming columns, deleting column, moving columns and other schema evolution were not
pursued due to lack of importance and lack of time.  Also, it appears a much more sophisticated
metadata would be needed to support them.
> The biggest issues for users have been adding new columns for ACID table (HIVE-11421
Support Schema evolution for ACID tables) and vectorization (HIVE-10598 Vectorization borks
when column is added to table).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message