hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <>
Subject [jira] [Created] (HIVE-17403) Fail concatenation for unmanaged tables
Date Tue, 29 Aug 2017 01:59:00 GMT
Prasanth Jayachandran created HIVE-17403:

             Summary: Fail concatenation for unmanaged tables
                 Key: HIVE-17403
             Project: Hive
          Issue Type: Bug
    Affects Versions: 1.3.0, 3.0.0, 2.4.0
            Reporter: Prasanth Jayachandran
            Assignee: Prasanth Jayachandran
            Priority: Blocker

ALTER TABLE .. CONCATENATE should fail if the table is not managed by hive. 

For unmanaged tables, file names can be anything. Hive has some assumptions about file names
which can result in data loss for unmanaged tables. 

Example of this is a table/partition having 2 different files files (part-m-00000__1417075294718
and part-m-00018__1417075294718). Although both are completely different files, hive thinks
these are files generated by separate instances of same task (because of failure or speculative
execution). Hive will end up removing this file

2017-08-28T18:19:29,516 WARN  [b27f10d5-d957-4695-ab2a-1453401793df main]: exec.Utilities
(:()) - Duplicate taskid file removed: file:/Users/table/part=20141120/.hive-staging_hive_2017-08-28_18-19-27_210_3381701454205724533-1/_tmp.-ext-10000/part-m-00018__1417075294718
with length 958510. Existing file: file:/Users/table/part=20141120/.hive-staging_hive_2017-08-28_18-19-27_210_3381701454205724533-1/_tmp.-ext-10000/part-m-00000__1417075294718
with length 1123116

DDL should restrict concatenation for unmanaged tables. 

This message was sent by Atlassian JIRA

View raw message