kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Guozhang Wang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (KAFKA-3778) Change the RocksDB implementation for KStream windowed aggregations, to not use “range queries” but multiple gets
Date Thu, 02 Jun 2016 22:49:59 GMT

     [ https://issues.apache.org/jira/browse/KAFKA-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Guozhang Wang updated KAFKA-3778:
---------------------------------
    Description: 
RocksDbWindowStore currently does not use caches, but its window segments implemented as RocksDbStore
does. However, its range query {{fetch(key, fromTime, toTime)}} will cause all its touched
segments' cache to be flushed. 

After KAFKA-3777, we should change its implementation for KStreamWindowAggregation / KStreamWindowReduce
to not use {{fetch}}, but just as multiple {{get}} calls on the underlying segments, one for
each affected window range.

  was:RocksDbWindowStore currently does not use caches (while RocksDbStore does). Enable caching
for RocksDbWindowStore. This might require changing the API from using range queries to using
multiple gets(), one for each sub-range.


> Change the RocksDB implementation for KStream windowed aggregations, to not use “range
queries” but multiple gets
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-3778
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3778
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: streams
>    Affects Versions: 0.10.1.0
>            Reporter: Eno Thereska
>             Fix For: 0.10.1.0
>
>
> RocksDbWindowStore currently does not use caches, but its window segments implemented
as RocksDbStore does. However, its range query {{fetch(key, fromTime, toTime)}} will cause
all its touched segments' cache to be flushed. 
> After KAFKA-3777, we should change its implementation for KStreamWindowAggregation /
KStreamWindowReduce to not use {{fetch}}, but just as multiple {{get}} calls on the underlying
segments, one for each affected window range.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message