ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Semyon Boikov <sboi...@gridgain.com>
Subject Re: Retrieving multiple keys with filtering
Date Tue, 29 Aug 2017 13:15:30 GMT
Hi,

If EntryProcessor returns null then null is not added in the result map.
But I agree that using invokeAll() will have a lot of unnecessary overhead.
Perhaps we need add new getAll method on API, otherwise best alternative is
use custom ComputeJob or affinityCall.

Thanks,
Semyon

On Tue, Aug 29, 2017 at 7:20 AM, Dmitriy Setrakyan <dsetrakyan@apache.org>
wrote:

> Andrey,
>
> I am not sure I understand. According to EntryProcessor API [1] you can
> chose to return nothing.
>
> Also, to my knowledge, you can still do parallel reads while executing the
> EntryProcessor. Perhaps other community members can elaborate on this.
>
> [1] https://static.javadoc.io/javax.cache/cache-api/1.0.0/
> index.html?javax/cache/processor/EntryProcessor.html
>
> D.
>
>
> On Mon, Aug 28, 2017 at 8:29 PM, Andrey Kornev <andrewkornev@hotmail.com>
> wrote:
>
>> Dmitriy,
>>
>>
>> It's good to be back! 😃 Glad to find Ignite community as vibrant
>> and thriving as ever!
>>
>> Speaking of invokeAll(), even if we ignore for a moment the overhead
>> associated with locking/unlocking a cache entry prior to passing it to the
>> EntryProcessor as well as the overhead associated with enlisting the
>> touched entries in a transaction, the bigger problem with using
>> invokeAll() for filtering is that EntryProcessor must return a value. I'm
>> not aware of any way to make EntryProcessor drop the entry from the
>> response. The only options is to use a null (or false) to indicate a
>> filtered out entry. In my specific case, I'll end up sending back a whole
>> bunch of nulls in the result map as I expect most of the keys to be
>> rejected by the filter.
>>
>> Overall, invokeAll() is not what one would call *efficient* (the key word
>> in my original question) way of filtering.
>>
>> Thanks!
>> Andrey
>>
>> ------------------------------
>> *From:* Dmitriy Setrakyan <dsetrakyan@apache.org>
>> *Sent:* Saturday, August 26, 2017 8:37 AM
>> *To:* user
>>
>> *Subject:* Re: Retrieving multiple keys with filtering
>>
>> Andrey,
>>
>> Good to hear from you. Long time no talk.
>>
>> I don't think invokeAll has only update semantics. You can definitely use
>> it just to look at the keys and return a result. Also, as you mentioned,
>> Ignite compute is a viable option as well.
>>
>> The reason that predicates were removed from the get methods is because
>> the API was becoming unwary, and also because JCache does not require it.
>>
>> D.
>>
>> On Thu, Aug 24, 2017 at 10:50 AM, Andrey Kornev <andrewkornev@hotmail.com
>> > wrote:
>>
>>> Well, I believe invokeAll() has "update" semantics and using it for
>>> read-only filtering of cache entries is probably not going to be efficient
>>> or even appropriate.
>>>
>>>
>>> I'm afraid the only viable option I'm left with is to use Ignite's
>>> Compute feature:
>>>
>>> - on the sender, group the keys by affinity.
>>>
>>> - send each group along with the filter predicate to their
>>> affinity nodes using IgniteCompute.
>>>
>>> - on each node, use getAll() to fetch the local keys and apply the
>>> filter.
>>>
>>> - on the sender node, collect the results of the compute jobs into a map.
>>>
>>>
>>> It's unfortunate that Ignite dropped that original API. What used to be
>>> a single API call is now a non-trivial algorithm and one have to worry
>>> about things like what happens if the grid topology changes while the
>>> compute jobs are executing, etc.
>>>
>>> Can anyone think of any other less complex/more robust approach?
>>>
>>> Thanks
>>> Andrey
>>>
>>> ------------------------------
>>> *From:* slava.koptilin <slava.koptilin@gmail.com>
>>> *Sent:* Thursday, August 24, 2017 9:03 AM
>>> *To:* user@ignite.apache.org
>>> *Subject:* Re: Retrieving multiple keys with filtering
>>>
>>> Hi Andrey,
>>>
>>> Yes, you are right. ScanQuery scans all entries.
>>> Perhaps, IgniteCache#invokeAll(keys, cacheEntryProcessor) with custom
>>> processor will work for you.
>>> https://ignite.apache.org/releases/2.1.0/javadoc/org/apache/
>>> ignite/IgniteCache.html#invokeAll(java.util.Set,%20org.
>>> apache.ignite.cache.CacheEntryProcessor,%20java.lang.Object...)
>>>
>>> Thanks!
>>>
>>>
>>>
>>> --
>>> View this message in context: http://apache-ignite-users.705
>>> 18.x6.nabble.com/Retrieving-multiple-keys-with-filtering-tp1
>>> 6391p16400.html
>>> Apache Ignite Users - Retrieving multiple keys with filtering
>>> <http://apache-ignite-users.70518.x6.nabble.com/Retrieving-multiple-keys-with-filtering-tp16391p16400.html>
>>> apache-ignite-users.70518.x6.nabble.com
>>> Retrieving multiple keys with filtering. Hello, I have a list of cache
>>> keys (up to a few hundred of them) and a filter predicate. I'd like to
>>> efficiently retrieve only those values that pass the...
>>>
>>>
>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>
>>
>>
>

Mime
View raw message