flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shannon Carey <sca...@expedia.com>
Subject State in external db (dynamodb)
Date Wed, 06 Apr 2016 05:55:53 GMT
Hi, new Flink user here!

I found a discussion on user@flink.apache.org about using DynamoDB as a sink. However, as
noted, sinks have an at-least-once guarantee so your operations must idempotent.

However, another way to go about this (and correct me if I'm wrong) is to write the state
to the external store via a custom State Backend. Since the state participates in checkpointing,
you don't have to worry about idempotency: every time state is checkpointed, overwrite the
value of that key.

We are starting a project with Flink, and we are interested in evicting the state from memory
once a TTL is reached during which no events have come in for that state. Subsequently, when
an event is processed, we must be able to quickly load up any evicted state. Does this sound
reasonable? We are considering using DynamoDB for our state backend because it seems like
all we will need is a key-value store. The only weakness of this is that if state gets older
than, say, 2 years we would like to get rid of it which might not be easy in DynamoDB. I don't
suppose Flink has any behind-the-scenes features that deal with getting rid of old state (either
evicting from memory or TTL/aging out entirely)?

Thanks for your time!
Shannon Carey
View raw message