kafka-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "John Roesler" <vvcep...@apache.org>
Subject Re: Role of CompositeReadOnlyKeyValueStore for point queries
Date Thu, 16 Jan 2020 16:52:22 GMT
Hey Navinder,

I agree this is not ideal. The answer is that until KIP-535, you wouldn't
necessarily know which partition you were querying, just the instance.

Now, though, the information is front-and-center, and we should
update the store() API to favor efficiently routing the query directly
to the right store instance, rather than iterating over all the stores.

I'd been privately planning to propose a new `KafkaStreams#store()`
method that lets you just request a store for a particular partition.
Reading your message, I think maybe this is what you meant, instead
of `get()`?

If you want to take it on, I'd be more than happy to take the lead on
reviewing your work.

One food for thought is that we just added a new overload:
`store(name, type, boolean includeStaleStores)`. Instead of adding
another new overload, we should replace the new overload with
one taking a config object.

Suggestion:
KafkaStreams#store(
  String storeName,
  QueryableStoreType<T> type,
  StoreQueryParams options
)

where:
StoreQueryParams {
  (enable/disable/get)IncludeStaleStores()
  (with/get)partitions(List<Integer>)
}

Although this isn't a DSL operation, please consider 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams+DSL+Grammar
when proposing new interfaces.

If you're _really_ fast, we could potentially get this in to 2.5:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=143428858
We'd basically have to call for a vote by Friday to have a chance of meeting the deadline.
Then, we could just replace the new overload instead of deprecating it.
But it's not such a huge deal to deprecate it, so no pressure.

Thanks for the astute observation!
-John

On Wed, Jan 15, 2020, at 22:50, Navinder Brar wrote:
> Hi all,
> Can someone explain to me the thoughts behind having 
> CompositeReadOnlyKeyValueStore. java class while serving data via APIs 
> in Kafka Streams. It fetches the list of stores for all the running 
> tasks on the machine and then looks for a key one by one in each of the 
> stores. When we already know the key belongs to a particular 
> partition(with the help of partitioner) we can just query that 
> particular partition's store right?
> I am thinking of overloading the get() function as get(int partition) 
> and sending just the store for that single partition from 
> QueryableStoreProvider.java so that all the stores are needed to be 
> iterated through to fetch a key.
> Regards,
> Navinder
>

Mime
View raw message