flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paris Carbone (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-2324) Rework partitioned state storage
Date Tue, 07 Jul 2015 13:18:04 GMT

    [ https://issues.apache.org/jira/browse/FLINK-2324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14616675#comment-14616675

Paris Carbone commented on FLINK-2324:

We cannot avoid a key-value store to make this efficient imho. There are several levelDB backed
distributed key value stores we can use to persist and retrieve partitioned state. There is
also HBase though if we want to stick to HDFS for this...

Any other opinions?

> Rework partitioned state storage
> --------------------------------
>                 Key: FLINK-2324
>                 URL: https://issues.apache.org/jira/browse/FLINK-2324
>             Project: Flink
>          Issue Type: Improvement
>            Reporter: Gyula Fora
>            Assignee: Gyula Fora
> Partitioned states are currently stored per-key in statehandles. This is alright for
in-memory storage but is very inefficient for HDFS. 
> The logic behind the current mechanism is that this approach provides a way to repartition
a state without fetching the data from the external storage and only manipulating handles.
> We should come up with a solution that can achieve both.

This message was sent by Atlassian JIRA

View raw message