kafka-jira mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matthias J. Sax (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (KAFKA-4499) Add "getAllKeys" API for querying windowed KTable stores
Date Mon, 25 Sep 2017 02:51:00 GMT

    [ https://issues.apache.org/jira/browse/KAFKA-4499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178466#comment-16178466

Matthias J. Sax commented on KAFKA-4499:

First, the description above is just a sketch/place-holder and the actual design must be done
in a KIP. For methods to be added: they should be added to {{ReadOnlyWindowStore}} ({{WindowStore}}
will automatically inherit them of course -- note, that {{WindowStore}} extends {{ReadOnlyWindowStore}},
but not the other way round). The actual implementation would be done in {{RocksDBWindowStore}}
and {{CachingWindowStore}} (of course other classes will be affected, too). It's not a trivial
change, if that was your question.

> Add "getAllKeys" API for querying windowed KTable stores
> --------------------------------------------------------
>                 Key: KAFKA-4499
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4499
>             Project: Kafka
>          Issue Type: Improvement
>          Components: streams
>            Reporter: Matthias J. Sax
>              Labels: needs-kip
> Currently, both {{KTable}} and windowed-{{KTable}} stores can be queried via IQ feature.
While {{ReadOnlyKeyValueStore}} (for {{KTable}} stores) provide method {{all()}} to scan the
whole store (ie, returns an iterator over all stored key-value pairs), there is no similar
API for {{ReadOnlyWindowStore}} (for windowed-{{KTable}} stores).
> This limits the usage of a windowed store, because the user needs to know what keys are
stored in order the query it. It would be useful to provide possible APIs like this (only
a rough sketch):
>  - {{keys()}} returns all keys available in the store (maybe together with available
time ranges)
>  - {{all(long timeFrom, long timeTo)}} that returns all window for a specific time range
>  - {{allLatest()}} that returns the latest window for each key
> Because this feature would require to scan multiple segments (ie, RockDB instances) it
would be quite inefficient with current store design. Thus, this feature also required to
redesign the underlying window store itself.
> Because this is a major change, a KIP (https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals)
is required. The KIP should cover the actual API design as well as the store refactoring.

This message was sent by Atlassian JIRA

View raw message