flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pluppens <...@git.apache.org>
Subject [GitHub] flink issue #5337: [FLINK-8484][flink-kinesis-connector] Ensure a Kinesis co...
Date Thu, 01 Feb 2018 10:43:24 GMT
Github user pluppens commented on the issue:

    Regarding the remark from @StephanEwen: perhaps it would be ok to re-use the `KinesisProxy`
to return a list of all shards and compare them to the `sequenceNumsToRestore` to prune any
shards that no longer exist? It would delay the restoration, but you'd be sure the state wouldn't
grow indefinitely (we were looking at around a 1000 closed shards with a 24 hour retention
period, so 365k per year - that's not going to end well). Another option would be to kick
off another task periodically to prune them, but that is likely to run into race conditions,
so doing it at the safe point of restoration would make more sense to me.


View raw message