hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saket Saurabh (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-14366) Conversion of a Non-ACID table to an ACID table produces non-unique primary keys
Date Thu, 28 Jul 2016 02:18:20 GMT

     [ https://issues.apache.org/jira/browse/HIVE-14366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Saket Saurabh updated HIVE-14366:
---------------------------------
    Description: 
When a Non-ACID table is converted to an ACID table, the primary key consisting of (original
transaction id, bucket_id, row_id) is not generated uniquely. Currently, the row_id is always
set to 0 for most rows. This leads to correctness issue for such tables.

Quickest way to reproduce is to add the following unit test to ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java

{code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid}
  @Test
  public void testOriginalReader() throws Exception {
    FileSystem fs = FileSystem.get(hiveConf);
    FileStatus[] status;

    // 1. Insert five rows to Non-ACID table.
    runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b) values(1,2),(3,4),(5,6),(7,8),(9,10)");

    // 2. Convert NONACIDORCTBL to ACID table.
    runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET TBLPROPERTIES ('transactional'='true')");

    // 3. Perform a major compaction.
    runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact 'MAJOR'");
    runWorker(hiveConf);

    // 4. Perform a delete.
    runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a = 1");

    // 5. Now do a projection should have (3,4) (5,6),(7,8),(9,10) only since (1,2) has been
deleted.
    List<String> rs = runStatementOnDriver("select a,b from " + Table.NONACIDORCTBL
+ " order by a,b");
    int[][] resultData = new int[][] {{3,4}, {5,6}, {7,8}, {9,10}};
    Assert.assertEquals(stringifyValues(resultData), rs);
  }
{code}

  was:
When a Non-ACID table is converted to an ACID table, the primary key consisting of (original
transaction id, bucket_id, row_id) is not generated uniquely. Currently, the row_id is always
set to 0 for most rows. This leads to correctness issue for such tables.

Quickest way to reproduce is to add the following unit test to ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java

{code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid}
  @Test
  public void testOriginalReader() throws Exception {
    FileSystem fs = FileSystem.get(hiveConf);
    FileStatus[] status;

    // 1. Insert five rows to Non-ACID table.
    runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b) values(1,2),(3,4),(5,6),(7,8),(9,10)");

    // 2. Convert NONACIDORCTBL to ACID table.
    runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET TBLPROPERTIES ('transactional'='true')");

    // 3. Perform a major compaction.
    runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact 'MAJOR'");
    runWorker(hiveConf);

    // 3. Perform a delete.
    runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a = 1");

    // Now do a projection should have (3,4) (5,6),(7,8),(9,10) only since (1,2) has been
deleted.
    List<String> rs = runStatementOnDriver("select a,b from " + Table.NONACIDORCTBL
+ " order by a,b");
    int[][] resultData = new int[][] {{3,4}, {5,6}, {7,8}, {9,10}};
    Assert.assertEquals(stringifyValues(resultData), rs);
  }
{code}


> Conversion of a Non-ACID table to an ACID table produces non-unique primary keys
> --------------------------------------------------------------------------------
>
>                 Key: HIVE-14366
>                 URL: https://issues.apache.org/jira/browse/HIVE-14366
>             Project: Hive
>          Issue Type: Bug
>          Components: Transactions
>            Reporter: Saket Saurabh
>
> When a Non-ACID table is converted to an ACID table, the primary key consisting of (original
transaction id, bucket_id, row_id) is not generated uniquely. Currently, the row_id is always
set to 0 for most rows. This leads to correctness issue for such tables.
> Quickest way to reproduce is to add the following unit test to ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java
> {code:title=ql/src/test/org/apache/hadoop/hive/ql/TestTxnCommands2.java|borderStyle=solid}
>   @Test
>   public void testOriginalReader() throws Exception {
>     FileSystem fs = FileSystem.get(hiveConf);
>     FileStatus[] status;
>     // 1. Insert five rows to Non-ACID table.
>     runStatementOnDriver("insert into " + Table.NONACIDORCTBL + "(a,b) values(1,2),(3,4),(5,6),(7,8),(9,10)");
>     // 2. Convert NONACIDORCTBL to ACID table.
>     runStatementOnDriver("alter table " + Table.NONACIDORCTBL + " SET TBLPROPERTIES ('transactional'='true')");
>     // 3. Perform a major compaction.
>     runStatementOnDriver("alter table "+ Table.NONACIDORCTBL + " compact 'MAJOR'");
>     runWorker(hiveConf);
>     // 4. Perform a delete.
>     runStatementOnDriver("delete from " + Table.NONACIDORCTBL + " where a = 1");
>     // 5. Now do a projection should have (3,4) (5,6),(7,8),(9,10) only since (1,2) has
been deleted.
>     List<String> rs = runStatementOnDriver("select a,b from " + Table.NONACIDORCTBL
+ " order by a,b");
>     int[][] resultData = new int[][] {{3,4}, {5,6}, {7,8}, {9,10}};
>     Assert.assertEquals(stringifyValues(resultData), rs);
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message