tephra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEPHRA-216) Invalid Transaction List Pruning will not proceed if there are empty transactional tables
Date Mon, 13 Feb 2017 18:55:41 GMT

    [ https://issues.apache.org/jira/browse/TEPHRA-216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15864209#comment-15864209
] 

ASF GitHub Bot commented on TEPHRA-216:
---------------------------------------

Github user gokulavasan commented on a diff in the pull request:

    https://github.com/apache/incubator-tephra/pull/34#discussion_r100868455
  
    --- Diff: tephra-hbase-compat-0.96/src/test/java/org/apache/tephra/hbase/txprune/InvalidListPruneTest.java
---
    @@ -341,6 +310,87 @@ public HTableInterface get() throws IOException {
         }
       }
     
    +  @Test
    +  public void testPruneEmptyTable() throws Exception {
    +    // Make sure that empty tables do not block the progress of pruning
    +
    +    // Create an empty table
    +    TableName txEmptyTable = TableName.valueOf("emptyPruneTestTable");
    +    HTable emptyHTable = createTable(txEmptyTable.getName(), new byte[][]{family}, false,
    +                                     Collections.singletonList(TestTransactionProcessor.class.getName()));
    +
    +    TransactionPruningPlugin transactionPruningPlugin = new TestTransactionPruningPlugin();
    +    transactionPruningPlugin.initialize(conf);
    +
    +    try {
    +      long now1 = System.currentTimeMillis();
    +      long inactiveTxTimeNow1 = (now1 - 150) * TxConstants.MAX_TX_PER_MS;
    +      long noPruneUpperBound = -1;
    +      long expectedPruneUpperBound1 = (now1 - 200) * TxConstants.MAX_TX_PER_MS;
    +      InMemoryTransactionStateCache.setTransactionSnapshot(
    +        new TransactionSnapshot(expectedPruneUpperBound1, expectedPruneUpperBound1, expectedPruneUpperBound1,
    +                                ImmutableSet.of(expectedPruneUpperBound1),
    +                                ImmutableSortedMap.<Long, TransactionManager.InProgressTx>of()));
    +      testUtil.compact(txEmptyTable, true);
    +      testUtil.compact(txDataTable1, true);
    +      // Since the write to prune table happens async, we need to sleep a bit before
checking the state of the table
    +      TimeUnit.SECONDS.sleep(2);
    +
    +      // fetch prune upper bound, there should be no prune upper bound since txEmptyTable
cannot be compacted
    +      long pruneUpperBound1 = transactionPruningPlugin.fetchPruneUpperBound(now1, inactiveTxTimeNow1);
    +      Assert.assertEquals(noPruneUpperBound, pruneUpperBound1);
    +      transactionPruningPlugin.pruneComplete(now1, noPruneUpperBound);
    +
    +      // Now flush the empty table, this will record the table region as empty, and then
pruning will continue
    +      testUtil.flush(txEmptyTable);
    +      // Since the write to prune table happens async, we need to sleep a bit before
checking the state of the table
    +      TimeUnit.SECONDS.sleep(2);
    +
    +      // fetch prune upper bound, again, this time it should work
    +      pruneUpperBound1 = transactionPruningPlugin.fetchPruneUpperBound(now1, inactiveTxTimeNow1);
    +      Assert.assertEquals(expectedPruneUpperBound1, pruneUpperBound1);
    +      transactionPruningPlugin.pruneComplete(now1, expectedPruneUpperBound1);
    +
    +      // Now add some data to the empty table
    +      // (adding data non-transactionally is okay too, we just need some data for the
compaction to run)
    +      emptyHTable.put(new Put(Bytes.toBytes(1)).add(family, qualifier, Bytes.toBytes(1)));
    +      emptyHTable.close();
    +
    +      // Now run another compaction on txDataTable1 with an updated tx snapshot
    +      long now2 = System.currentTimeMillis();
    +      long inactiveTxTimeNow2 = (now2 - 150) * TxConstants.MAX_TX_PER_MS;
    +      long expectedPruneUpperBound2 = (now2 - 200) * TxConstants.MAX_TX_PER_MS;
    +      InMemoryTransactionStateCache.setTransactionSnapshot(
    +        new TransactionSnapshot(expectedPruneUpperBound2, expectedPruneUpperBound2, expectedPruneUpperBound2,
    +                                ImmutableSet.of(expectedPruneUpperBound2),
    +                                ImmutableSortedMap.<Long, TransactionManager.InProgressTx>of()));
    +      testUtil.flush(txEmptyTable);
    +      testUtil.compact(txDataTable1, true);
    +      // Since the write to prune table happens async, we need to sleep a bit before
checking the state of the table
    +      TimeUnit.SECONDS.sleep(2);
    +
    +      // Running a prune now should still return min(inactiveTxTimeNow1, expectedPruneUpperBound1)
since
    +      // txEmptyTable is no longer empty. This information is returned since the txEmptyTable
was recorded as being
    +      // empty in the previous run with inactiveTxTimeNow1
    +      long pruneUpperBound2 = transactionPruningPlugin.fetchPruneUpperBound(now2, inactiveTxTimeNow2);
    +      Assert.assertEquals(inactiveTxTimeNow1, pruneUpperBound2);
    +      transactionPruningPlugin.pruneComplete(now2, expectedPruneUpperBound1);
    +
    +      // However, after compacting txEmptyTable we should get the latest upper bound
    +      testUtil.flush(txEmptyTable);
    +      testUtil.compact(txEmptyTable, true);
    +      // Since the write to prune table happens async, we need to sleep a bit before
checking the state of the table
    +      TimeUnit.SECONDS.sleep(2);
    +      pruneUpperBound2 = transactionPruningPlugin.fetchPruneUpperBound(now2, inactiveTxTimeNow2);
    +      Assert.assertEquals(expectedPruneUpperBound2, pruneUpperBound2);
    +      transactionPruningPlugin.pruneComplete(now2, expectedPruneUpperBound2);
    +    } finally {
    +      transactionPruningPlugin.destroy();
    +      hBaseAdmin.disableTable(txEmptyTable);
    +      hBaseAdmin.deleteTable(txEmptyTable);
    +    }
    +    }
    --- End diff --
    
    Fix alignment


> Invalid Transaction List Pruning will not proceed if there are empty transactional tables
> -----------------------------------------------------------------------------------------
>
>                 Key: TEPHRA-216
>                 URL: https://issues.apache.org/jira/browse/TEPHRA-216
>             Project: Tephra
>          Issue Type: Sub-task
>    Affects Versions: 0.11.0-incubating
>            Reporter: Gokul Gunasekaran
>            Assignee: Poorna Chandra
>
> Since empty table/regions might not undergo major compaction, the prune upper bound of
these regions will never have a value and thus will limit the progress of invalid list transaction
pruning.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message