flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aljoscha Krettek (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-3692) Develop a Kafka state backend
Date Mon, 04 Apr 2016 08:25:25 GMT

    [ https://issues.apache.org/jira/browse/FLINK-3692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15223817#comment-15223817
] 

Aljoscha Krettek commented on FLINK-3692:
-----------------------------------------

This is definitely interesting. How would this work for multiple checkpoints. In Flink, the
savepoints feature allows to make a special checkpoint (a savepoint) that can be restored
later. You can have several of those, Using HDFS this is very easy to achieve because the
snapshots are isolated. With Kafka (and especially with log compaction) you can only have
one view of the state for all keys, or am I wrong here?

> Develop a Kafka state backend
> -----------------------------
>
>                 Key: FLINK-3692
>                 URL: https://issues.apache.org/jira/browse/FLINK-3692
>             Project: Flink
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Elias Levy
>
> Flink clusters usually consume of a Kafka cluster.  It simplify operations if Flink could
store its state checkpoints in Kafka.  This should be possibly using different topics to write
to, partitioning appropriately, and using compacted topics.  This would avoid the need to
run an HDFS cluster just to store Flink checkpoints.
> For inspiration you may want to take a look at how Samza checkpoints a task's local state
to a Kafka topic, and how the newer Kafka consumers checkpoint their offsets to Kafka.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message