hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <>
Subject [jira] [Updated] (HIVE-8368) compactor is improperly writing delete records in base file
Date Tue, 07 Oct 2014 18:32:33 GMT


Alan Gates updated HIVE-8368:
    Attachment: HIVE-8368.patch

This patch adds a new isDelete method to AcidInputFormat.RawReader.  The compactor then calls
isDelete when getting records from RawReader and then throws them out if it is doing a major

A note on adding isDelete.  I think it would be better to change to not return
deleted records when the option compact is set to true.  Two things prevented me from doing
this.  One, while I think no one ever needs deletes in the compact case I might be wrong.
 Two, the loop that does the merge and returns the next record in is complicated
code, and changing it looked very error prone.

> compactor is improperly writing delete records in base file
> -----------------------------------------------------------
>                 Key: HIVE-8368
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>    Affects Versions: 0.14.0
>            Reporter: Alan Gates
>            Assignee: Alan Gates
>            Priority: Critical
>             Fix For: 0.14.0
>         Attachments: HIVE-8368.patch
> When the compactor reads records from the base and deltas, it is not properly dropping
delete records.  This leads to oversized base files, and possibly to wrong query results.

This message was sent by Atlassian JIRA

View raw message