flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From StefanRRichter <...@git.apache.org>
Subject [GitHub] flink pull request #4745: [FLINK-7720] [state backends] Centralize creation ...
Date Thu, 28 Sep 2017 19:26:09 GMT
GitHub user StefanRRichter opened a pull request:


    [FLINK-7720] [state backends] Centralize creation of backends and state related resources

    ## What is the purpose of the change
    This PR combines [FLINK-7719] and [FLINK-7720], which are both related to our efforts
to introduce task recovery from local state. [FLINK-7719] makes the checkpoint id part of
the task deployment descriptior, which is required so that tasks can figure our the right
state for local recovery. [FLINK-7720] introduces state manager components on task manager,
task, and stream task level that build upon each other. The purpose of them is unifying the
creation of state related elements such as backends. State reporting now goes through those
state managers.
    ## Brief change log
      - *JobManagerTaskRestore carries the checkpoint state handles and checkpoint id that
the JM wants the task to restore*
      - *Introduced TaskExecutorLocalStateStoresManager (mostly as placeholder/mock), TaskLocalStateStore
(currently only a placeholder/ mock), TaskStateManager, and StreamTaskStateManager. Those
build a hierarchy for state reporting and recovery.*
      - *All state related elements are now created by the StreamTaskStateManager and owned
by the stream operator.*
    ## Verifying this change
    This change is already covered by existing tests and is currently mostly a refactoring
and preparation for future extension. New test cases have been added for implementations of
new components. There should currently no observable changes for users.
    ## Does this pull request potentially affect one of the following parts:
      - Dependencies (does it add or upgrade a dependency): (no)
      - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes)
      - The serializers: (no)
      - The runtime per-record code paths (performance sensitive): (no)
      - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing,
Yarn/Mesos, ZooKeeper: (yes)
    ## Documentation
      - Does this pull request introduce a new feature? (no)

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/StefanRRichter/flink state-manager-pull

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #4745
commit e20958a2b54856cba353ea52bc603a16bee0611c
Author: Stefan Richter <s.richter@data-artisans.com>
Date:   2017-08-21T12:31:38Z

    [FLINK-7719] [checkpoints] Send checkpoint id to task as part of deployment descriptor
when resuming

commit c27c692a36305c79329e01031a43ae880039c9fd
Author: Stefan Richter <s.richter@data-artisans.com>
Date:   2017-09-27T13:10:52Z

    [FLINK-7720] [checkpoints] Centralize creation of backends and state related resources



View raw message