flink-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elias Levy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (FLINK-3692) Develop a Kafka state backend
Date Mon, 04 Apr 2016 01:45:25 GMT
Elias Levy created FLINK-3692:

             Summary: Develop a Kafka state backend
                 Key: FLINK-3692
                 URL: https://issues.apache.org/jira/browse/FLINK-3692
             Project: Flink
          Issue Type: New Feature
          Components: Core
            Reporter: Elias Levy

Flink clusters usually consume of a Kafka cluster.  It simplify operations if Flink could
store its state checkpoints in Kafka.  This should be possibly using different topics to write
to, partitioning appropriately, and using compacted topics.  This would avoid the need to
run an HDFS cluster just to store Flink checkpoints.

For inspiration you may want to take a look at how Samza checkpoints a task's local state
to a Kafka topic, and how the newer Kafka consumers checkpoint their offsets to Kafka.

This message was sent by Atlassian JIRA

View raw message