tephra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "ASF GitHub Bot (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (TEPHRA-241) Introduce a way to limit the size of a transaction
Date Thu, 31 Aug 2017 21:52:00 GMT

    [ https://issues.apache.org/jira/browse/TEPHRA-241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16149664#comment-16149664

ASF GitHub Bot commented on TEPHRA-241:

GitHub user anew opened a pull request:


    [TEPHRA-241] Introduce a way to limit the size of a transaction

    - first commit introduces four new configurations for change set size limits, enforces
these limits in a new method canCommitOrThrow() (we keep canCommit() for compatibility), and
uses the new method throughout the higher level classes and tests
    - second commit refactors the transaction system test cases: Don't start a new TxManager
for each test method, have a shared common configuration across subclasses, move the checkpoint
test to the base class such that it also gets tested with Thrift
    - third commit adds a new test case for size limit validation
    Thrift changes are backward-compatible (we keep the existing methods the same), and so
are the TransactionSystemClient changes. 

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/anew/incubator-tephra tephra-241

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #48
commit 7227bcef076ccf762b78bf299003c373e7e73814
Author: anew <anew@apache.org>
Date:   2017-08-31T21:43:32Z

    add new config properties and a new method canCommitOrThrow() that validates the size
of change sets

commit 4ad0f924c34ad4aea35443c8cd10bd81cf24eeb8
Author: anew <anew@apache.org>
Date:   2017-08-31T21:43:56Z

    refactor transaction system test

commit f3a1e786ae62f6bf05043c6fd770338361e615bc
Author: anew <anew@apache.org>
Date:   2017-08-31T21:44:32Z

    add new test for change set size validation


> Introduce a way to limit the size of a transaction
> --------------------------------------------------
>                 Key: TEPHRA-241
>                 URL: https://issues.apache.org/jira/browse/TEPHRA-241
>             Project: Tephra
>          Issue Type: Improvement
>          Components: api, manager
>    Affects Versions: 0.12.0-incubating
>            Reporter: Andreas Neumann
>            Assignee: Andreas Neumann
>             Fix For: 0.13.0-incubating
> When clients perform a huge number of writes in a short transaction, that can result
in huge change sets. For example, if a client performs 10M writes and sends that change set
over, that can easily be 1GB large. The transaction manager will keep this in memory. It will
also write this as an edit to the transaction log.
> Assume it runs out of memory because the change set is too large. It crashes and when
it restarts, it will replay the log, load that huge change set again, and crash again. 
> To prevent this kind of systemic failure, and to encourage developers to use long transactions
when performing many writes, we can introduce two new properties in the configuration:
> - change set warn threshold: if a change set exceeds this size, a warning is logged.

> - change set reject threshold: if a change set exceeds this size, it is rejected (canCommit
will throw an exception) and that will fail the transaction.
> Both thresholds should be Long.MAX_VALUE by default, to preserve existing behavior after

This message was sent by Atlassian JIRA

View raw message