flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aljoscha Krettek <aljos...@apache.org>
Subject Re: Kafka state backend?
Date Thu, 07 Apr 2016 10:06:41 GMT
Hi Zach,
I'm afraid someone already beat you to it :-)

In the issue we touch on some of the difficulties with this that stem from
the differences in the guarantees that Flink and Samza try to give.


On Tue, 5 Apr 2016 at 22:24 Zach Cox <zcox522@gmail.com> wrote:

> Hi - as clarified in another thread [1] stateful operators store all of
> their current state in the backend on each checkpoint. Just curious if
> Kafka topics with log compaction have ever been considered as a possible
> state backend?
> Samza [2] uses RocksDB as a local state store, with all writes also going
> to a log-compacted Kafka topic for persistence. This seems like it might
> also be a good alternative backend in Flink for jobs with large amounts of
> long-lasting state. You would give up some throughput (due to Kafka
> producer writes) but there would be almost nothing to do on checkpoints.
> Just wanted to propose the idea and see if it has already been discussed,
> or maybe I'm missing some reasons why it would be a bad idea.
> Thanks,
> Zach
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Checkpoint-state-stored-in-backend-and-deleting-old-checkpoint-state-td5927.html
> [2]
> http://samza.apache.org/learn/documentation/0.10/container/state-management.html#local-state-in-samza

View raw message