flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Syinchwun Leo (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (FLINK-4266) Remote Database Statebackend
Date Tue, 17 Jan 2017 04:10:26 GMT

    [ https://issues.apache.org/jira/browse/FLINK-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15824927#comment-15824927
] 

Syinchwun Leo edited comment on FLINK-4266 at 1/17/17 4:09 AM:
---------------------------------------------------------------

We also meet this problem,and plan to using incremental method. In current master branch,
there are four window state: ValueState, ListState, ReduingState and FoldingState. In ReducingState
and FoldingState, window only keeps intermediate results calculated by UDF reducing function
or folding function. When a element coming, the related intermediate result is recomputed
again by applying the UDF function. In my opinion it is costly to record all the changes in
realtime, what's more, we have to scan the pipeline reversely for the purpose of recovery
. In current sliding window's saving method, there is a big drawback: an element may be kept
in multiple windows, when checkpointing, all these backups are export to persistent stores,
it's time and storage consuming. we have modify the saving method of sliding window (including
SlidingProcessingTimeWindow and SlidingEventTimeWIndow), and it is testing. You can keep in
touch with me(liuxch@mail.ustc.edu.cn), and we can develop this feature in cooperation.


was (Author: syinchwunleo):
We also meet this problem,and plan to using incremental method. In currently master branch,
there are four window state: ValueState, ListState, ReduingState and FoldingState. In ReducingState
and FoldingState, window only keeps intermediate results calculated by UDF reducing function
or folding function. When a element coming, the related intermediate result is recomputed
again by applying the UDF function. In my opinion it is costly to record all the changes in
realtime, what's more, we have to scan the pipeline reversely for the purpose of recovery
. In current sliding window's saving method, there is a big drawback: an element may be kept
in multiple windows, when checkpointing, all these backups are export to persistent stores,
it's time and storage consuming. we have modify the saving method of sliding window (including
SlidingProcessingTimeWindow and SlidingEventTimeWIndow), and it is testing. You can keep in
touch with me(liuxch@mail.ustc.edu.cn), and we can develop this feature in cooperation.

> Remote Database Statebackend
> ----------------------------
>
>                 Key: FLINK-4266
>                 URL: https://issues.apache.org/jira/browse/FLINK-4266
>             Project: Flink
>          Issue Type: New Feature
>          Components: State Backends, Checkpointing
>    Affects Versions: 1.2.0
>            Reporter: Chen Qin
>            Priority: Minor
>
> Current FileSystem statebackend limits whole state size to disk space. Dealing with scale
out checkpoint states beyond local disk space such as long running task that hold window content
for long period of time. Pipelines needs to split out states to durable remote storage even
replicated to different data centers.
> We draft a design that leverage checkpoint id as mono incremental logic timestamp and
perform range query to get evicited state k/v. we also introduce checkpoint time commit and
eviction threshold that reduce "hot states" hitting remote db per every update between adjacent
checkpoints by tracking update logs and merge them, do batch updates only when checkpoint;
lastly, we are looking for eviction policy that can identify "hot keys" in k/v state and lazy
load those "cold keys" from remote storage(e.g Cassandra).
> For now, we don't have good story regarding to data retirement. We might have to keep
forever until manually run command and clean per job related state data. Some of features
might related to incremental checkpointing feature, we hope to align with effort there also.
> Welcome comments, I will try to put a draft design doc after gathering some feedback.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message