kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4212) Add a key-value store that is a TTL persistent cache
Date Wed, 28 Sep 2016 21:22:21 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530909#comment-15530909
] 

Guozhang Wang commented on KAFKA-4212:
--------------------------------------

Thanks for the detailed description Elias, I think a {{PersistentLRUCacheStore}} would be
a good add to fit this scenario. 

> Add a key-value store that is a TTL persistent cache
> ----------------------------------------------------
>
>                 Key: KAFKA-4212
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4212
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>    Affects Versions: 0.10.0.1
>            Reporter: Elias Levy
>            Assignee: Guozhang Wang
>
> Some jobs needs to maintain as state a large set of key-values for some period of time.
 I.e. they need to maintain a TTL cache of values potentially larger than memory. 
> Currently Kafka Streams provides non-windowed and windowed key-value stores.  Neither
is an exact fit to this use case.  
> The {{RocksDBStore}}, a {{KeyValueStore}}, stores one value per key as required, but
does not support expiration.  The TTL option of RocksDB is explicitly not used.
> The {{RocksDBWindowsStore}}, a {{WindowsStore}}, can expire items via segment dropping,
but it stores multiple items per key, based on their timestamp.  But this store can be repurposed
as a cache by fetching the items in reverse chronological order and returning the first item
found.
> KAFKA-2594 introduced a fixed-capacity in-memory LRU caching store, but here we desire
a variable-capacity memory-overflowing TTL caching store.
> Although {{RocksDBWindowsStore}} can be repurposed as a cache, it would be useful to
have an official and proper TTL cache API and implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message