tephra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEPHRA-241) Introduce a way to limit the size of a transaction
Date Thu, 31 Aug 2017 21:53:00 GMT

    [ https://issues.apache.org/jira/browse/TEPHRA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149667#comment-16149667
] 

ASF GitHub Bot commented on TEPHRA-241:
---------------------------------------

Github user anew commented on a diff in the pull request:

    https://github.com/apache/incubator-tephra/pull/48#discussion_r136457929
  
    --- Diff: tephra-core/src/test/java/org/apache/tephra/TransactionManagerTest.java ---
    @@ -54,115 +54,35 @@ protected TransactionStateStorage getStateStorage() {
         return txStateStorage;
       }
     
    -  @Before
    -  public void before() {
    +  @BeforeClass
    +  public static void beforeClass() {
    +    conf = getCommonConfiguration();
         conf.setInt(TxConstants.Manager.CFG_TX_CLEANUP_INTERVAL, 0); // no cleanup thread
    -    conf.setInt(TxConstants.Manager.CFG_TX_MAX_TIMEOUT, (int) TimeUnit.DAYS.toSeconds(5));
// very long limit
         // todo should create two sets of tests, one with LocalFileTxStateStorage and one
with InMemoryTxStateStorage
         txStateStorage = new InMemoryTransactionStateStorage();
         txManager = new TransactionManager
           (conf, txStateStorage, new TxMetricsCollector());
         txManager.startAndWait();
       }
     
    -  @After
    -  public void after() {
    +  @AfterClass
    +  public static void afterClass() {
         txManager.stopAndWait();
       }
     
    -  @Test
    --- End diff --
    
    this test moved to TransactionSystemTest without change.



> Introduce a way to limit the size of a transaction
> --------------------------------------------------
>
>                 Key: TEPHRA-241
>                 URL: https://issues.apache.org/jira/browse/TEPHRA-241
>             Project: Tephra
>          Issue Type: Improvement
>          Components: api, manager
>    Affects Versions: 0.12.0-incubating
>            Reporter: Andreas Neumann
>            Assignee: Andreas Neumann
>             Fix For: 0.13.0-incubating
>
>
> When clients perform a huge number of writes in a short transaction, that can result
in huge change sets. For example, if a client performs 10M writes and sends that change set
over, that can easily be 1GB large. The transaction manager will keep this in memory. It will
also write this as an edit to the transaction log.
> Assume it runs out of memory because the change set is too large. It crashes and when
it restarts, it will replay the log, load that huge change set again, and crash again. 
> To prevent this kind of systemic failure, and to encourage developers to use long transactions
when performing many writes, we can introduce two new properties in the configuration:
> - change set warn threshold: if a change set exceeds this size, a warning is logged.

> - change set reject threshold: if a change set exceeds this size, it is rejected (canCommit
will throw an exception) and that will fail the transaction.
> Both thresholds should be Long.MAX_VALUE by default, to preserve existing behavior after
upgrade. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message