Mailing-List: contact issues-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Date: Wed, 31 May 2017 12:38:11 +0000 (UTC)
From: "Robert Metzger (JIRA)" <jira@apache.org>
To: issues@flink.apache.org
Message-ID: <JIRA.13041759.1486657431000.330847.1496234291629@Atlassian.JIRA>
In-Reply-To: <JIRA.13041759.1486657431000@Atlassian.JIRA>
References: <JIRA.13041759.1486657431000@Atlassian.JIRA> <JIRA.13041759.1486657431767@jira-lw-us.apache.org>
Subject: [jira] [Updated] (FLINK-5763) Make savepoints self-contained and
 relocatable
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
archived-at: Wed, 31 May 2017 12:38:20 -0000


     [ https://issues.apache.org/jira/browse/FLINK-5763?page=3Dcom.atlassia=
n.jira.plugin.system.issuetabpanels:all-tabpanel ]

Robert Metzger updated FLINK-5763:
----------------------------------
    Fix Version/s:     (was: 1.3.0)
                   1.4.0

> Make savepoints self-contained and relocatable
> ----------------------------------------------
>
>                 Key: FLINK-5763
>                 URL: https://issues.apache.org/jira/browse/FLINK-5763
>             Project: Flink
>          Issue Type: Improvement
>          Components: State Backends, Checkpointing
>            Reporter: Ufuk Celebi
>             Fix For: 1.4.0
>
>
> After a user has triggered a savepoint, a single savepoint file will be r=
eturned as a handle to the savepoint. A savepoint to {{<target>}} creates a=
 savepoint file like {{<target>/savepoint-<randomSuffix>}}.
> This file contains the metadata of the corresponding checkpoint, but not =
the actual program state. While this works well for short term management (=
pause-and-resume a job), it makes it hard to manage savepoints over longer =
periods of time.
> h4. Problems
> h5. Scattered Checkpoint Files
> For file system based checkpoints (FsStateBackend, RocksDBStateBackend) t=
his results in the savepoint referencing files from the checkpoint director=
y (usually different than <target>). For users, it is virtually impossible =
to tell which checkpoint files belong to a savepoint and which are lingerin=
g around. This can easily lead to accidentally invalidating a savepoint by =
deleting checkpoint files.
> h5. Savepoints Not Relocatable
> Even if a user is able to figure out which checkpoint files belong to a s=
avepoint, moving these files will invalidate the savepoint as well, because=
 the metadata file references absolute file paths.
> h5. Forced to Use CLI for Disposal
> Because of the scattered files, the user is in practice forced to use Fli=
nk=E2=80=99s CLI to dispose a savepoint. This should be possible to handle =
in the scope of the user=E2=80=99s environment via a file system delete ope=
ration.
> h4. Proposal
> In order to solve the described problems, savepoints should contain all t=
heir state, both metadata and program state, inside a single directory. Fur=
thermore the metadata must only hold relative references to the checkpoint =
files. This makes it obvious which files make up the state of a savepoint a=
nd it is possible to move savepoints around by moving the savepoint directo=
ry.
> h5. Desired File Layout
> Triggering a savepoint to {{<target>}} creates a directory as follows:
> {code}
> <target>/savepoint-<jobId>-<randomSuffix>
>   +-- _metadata
>   +-- data-<randomSuffix> [1 or more]
> {code}
> We include the JobID in the savepoint directory name in order to give som=
e hints about which job a savepoint belongs to.
> h5. CLI
> - Trigger: When triggering a savepoint to {{<target>}} the savepoint dire=
ctory will be returned as the handle to the savepoint.
> - Restore: Users can restore by pointing to the directory or the _metadat=
a file. The data files should be required to be in the same directory as th=
e _metadata file.
> - Dispose: The disposal command should be deprecated and eventually remov=
ed. While deprecated, disposal can happen by specifying the directory or th=
e _metadata file (same as restore).


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)