hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-8368) compactor is improperly writing delete records in base file
Date Tue, 07 Oct 2014 18:32:33 GMT

     [ https://issues.apache.org/jira/browse/HIVE-8368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Alan Gates updated HIVE-8368:
-----------------------------
    Attachment: HIVE-8368.patch

This patch adds a new isDelete method to AcidInputFormat.RawReader.  The compactor then calls
isDelete when getting records from RawReader and then throws them out if it is doing a major
compaction.

A note on adding isDelete.  I think it would be better to change RawReader.next to not return
deleted records when the option compact is set to true.  Two things prevented me from doing
this.  One, while I think no one ever needs deletes in the compact case I might be wrong.
 Two, the loop that does the merge and returns the next record in RawReader.next is complicated
code, and changing it looked very error prone.

> compactor is improperly writing delete records in base file
> -----------------------------------------------------------
>
>                 Key: HIVE-8368
>                 URL: https://issues.apache.org/jira/browse/HIVE-8368
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 0.14.0
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>            Priority: Critical
>             Fix For: 0.14.0
>
>         Attachments: HIVE-8368.patch
>
>
> When the compactor reads records from the base and deltas, it is not properly dropping
delete records.  This leads to oversized base files, and possibly to wrong query results.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message