hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karen Coppage (Jira)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-21266) Don't run cleaner if compaction is skipped (issue with single delta file)
Date Fri, 29 Nov 2019 10:00:18 GMT

     [ https://issues.apache.org/jira/browse/HIVE-21266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Karen Coppage updated HIVE-21266:
---------------------------------
    Attachment: HIVE-21266.02.patch
        Status: Patch Available  (was: Open)

> Don't run cleaner if compaction is skipped (issue with single delta file)
> -------------------------------------------------------------------------
>
>                 Key: HIVE-21266
>                 URL: https://issues.apache.org/jira/browse/HIVE-21266
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Transactions
>    Affects Versions: 4.0.0
>            Reporter: Eugene Koifman
>            Assignee: Karen Coppage
>            Priority: Major
>         Attachments: HIVE-21266.01.patch, HIVE-21266.02.patch, HIVE-21266.02.patch
>
>
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java#L353-L357]
>  
> {noformat}
> if ((deltaCount + (dir.getBaseDirectory() == null ? 0 : 1)) + origCount <= 1) {
>       LOG.debug("Not compacting {}; current base is {} and there are {} deltas and {}
originals", sd.getLocation(), dir
>           .getBaseDirectory(), deltaCount, origCount);
>       return;
>     }
>  {noformat}
> Is problematic.
> Suppose you have 1 delta file from streaming ingest: {{delta_11_20}} where {{txnid:13}}
was aborted.  The code above will not rewrite the delta (which drops anything that belongs
to the aborted txn) and transition the compaction to "ready_for_cleaning" state which will
drop the metadata about the aborted txn in {{markCleaned()}}.  Now aborted data will come
back as committed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message