tephra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEPHRA-244) Invalid tx pruning does not handle deletion of tables well
Date Mon, 11 Sep 2017 05:14:00 GMT

    [ https://issues.apache.org/jira/browse/TEPHRA-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16160718#comment-16160718
] 

ASF GitHub Bot commented on TEPHRA-244:
---------------------------------------

Github user anew commented on a diff in the pull request:

    https://github.com/apache/incubator-tephra/pull/55#discussion_r137981077
  
    --- Diff: tephra-hbase-compat-1.1-base/src/test/java/org/apache/tephra/hbase/txprune/InvalidListPruneTest.java
---
    @@ -384,7 +384,67 @@ public void testPruneEmptyTable() throws Exception {
           hBaseAdmin.disableTable(txEmptyTable);
           hBaseAdmin.deleteTable(txEmptyTable);
         }
    +  }
    +
    +  @Test
    +  public void testPruneTransientTable() throws Exception {
    +    // Make sure that transient tables do not block the progress of pruning
    +
    +    // Create a temp table
    +    TableName txTempTable = TableName.valueOf("tempTable");
    +    createTable(txTempTable.getName(), new byte[][]{family}, false,
    +                Collections.singletonList(TestTransactionProcessor.class.getName()));
    +
    +    TableName txDataTable2 = null;
    +
    +    TransactionPruningPlugin transactionPruningPlugin = new TestTransactionPruningPlugin();
    +    transactionPruningPlugin.initialize(conf);
    +
    +    try {
    +      long now1 = System.currentTimeMillis();
    +      long inactiveTxTimeNow1 = (now1 - 150) * TxConstants.MAX_TX_PER_MS;
    +      long noPruneUpperBound = -1;
    +      long expectedPruneUpperBound1 = (now1 - 200) * TxConstants.MAX_TX_PER_MS;
    +      InMemoryTransactionStateCache.setTransactionSnapshot(
    +        new TransactionSnapshot(expectedPruneUpperBound1, expectedPruneUpperBound1, expectedPruneUpperBound1,
    +                                ImmutableSet.of(expectedPruneUpperBound1),
    +                                ImmutableSortedMap.<Long, TransactionManager.InProgressTx>of()));
    +
    +      // fetch prune upper bound, there should be no prune upper bound since nothing
has been compacted yet.
    +      // This run is only to store the initial set of regions
    +      long pruneUpperBound1 = transactionPruningPlugin.fetchPruneUpperBound(now1, inactiveTxTimeNow1);
    +      Assert.assertEquals(noPruneUpperBound, pruneUpperBound1);
    +      transactionPruningPlugin.pruneComplete(now1, noPruneUpperBound);
    +
    +      // Now delete the transient table
    +      hBaseAdmin.disableTable(txTempTable);
    +      hBaseAdmin.deleteTable(txTempTable);
    +
    +      // Compact the data table now
    +      testUtil.compact(txDataTable1, true);
    +      // Since the write to prune table happens async, we need to sleep a bit before
checking the state of the table
    +      TimeUnit.SECONDS.sleep(2);
    +
    +      // Create a new table that will not be compacted
    +      txDataTable2 = TableName.valueOf("invalidListPruneTestTable2");
    +      createTable(txDataTable2.getName(), new byte[][]{family}, false,
    +                  Collections.singletonList(TestTransactionProcessor.class.getName()));
    +
    +      // fetch prune upper bound, there should be a prune upper bound even though txTempTable
does not exist anymore,
    +      // and txDataTable2 has not been compacted/flushed yet
    +      long now2 = System.currentTimeMillis();
    +      long inactiveTxTimeNow2 = (now1 - 150) * TxConstants.MAX_TX_PER_MS;
    +      long pruneUpperBound2 = transactionPruningPlugin.fetchPruneUpperBound(now2, inactiveTxTimeNow2);
    +      Assert.assertEquals(expectedPruneUpperBound1, pruneUpperBound2);
    +      transactionPruningPlugin.pruneComplete(now2, expectedPruneUpperBound1);
    +    } finally {
    +      transactionPruningPlugin.destroy();
    +      if (txDataTable2 != null) {
    --- End diff --
    
    you should also do this for txTempTable. The test case deletes it, but if it fails before
that, the table never gets deleted.


> Invalid tx pruning does not handle deletion of tables well
> ----------------------------------------------------------
>
>                 Key: TEPHRA-244
>                 URL: https://issues.apache.org/jira/browse/TEPHRA-244
>             Project: Tephra
>          Issue Type: Bug
>          Components: core, manager
>    Affects Versions: 0.12.0-incubating
>            Reporter: Andreas Neumann
>            Assignee: Poorna Chandra
>             Fix For: 0.13.0-incubating
>
>
> Suppose an application regularly creates and deletes tables (for example, temporary tables).
In such a scenario, there will always be such a temporary table when pruning runs, and its
regions will be recorded for that time. However, the region will be deleted before it ever
compacts or flushes, and it will never record prune information. Because all prune times have
such regions, there will never be a set of transactional regions that all have prune info,
and pruning will never happen. 
> The fix is to exclude deleted tables from the list of regions at each time. This is all
regions of deleted tables (tables that do not exist any more), not deleted regions: a region
may disappear due to a split and its data will be in new regions. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message