ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: Retrieving multiple keys with filtering
Date Mon, 04 Sep 2017 03:39:44 GMT
Semyon,

Can you please clarify this. Do we allow concurrent reads while invokeAll
or invoke is executed?

D.

On Tue, Aug 29, 2017 at 11:59 AM, Andrey Kornev <andrewkornev@hotmail.com>
wrote:

> Ah, yes! Thank you, Semyon! According to invokeAll() javadocs "No mappings
> will be returned for EntryProcessors that return a null value for a key." I
> should read JCache javadocs more carefully next time. :)
>
>
> Still, the processor is invoked while a monitor is held on the cache entry
> being processed, which is of course unnecessary in a read-only case like
> the one we're discussing in this thread...
>
>
> I guess I'm stuck with the Compute-based approach for now. :(
>
> Thanks!
> Andrey
>
> ------------------------------
> *From:* Semyon Boikov <sboikov@gridgain.com>
> *Sent:* Tuesday, August 29, 2017 6:15 AM
>
> *To:* user@ignite.apache.org
> *Subject:* Re: Retrieving multiple keys with filtering
>
> Hi,
>
> If EntryProcessor returns null then null is not added in the result map.
> But I agree that using invokeAll() will have a lot of unnecessary overhead.
> Perhaps we need add new getAll method on API, otherwise best alternative is
> use custom ComputeJob or affinityCall.
>
> Thanks,
> Semyon
>
> On Tue, Aug 29, 2017 at 7:20 AM, Dmitriy Setrakyan <dsetrakyan@apache.org>
> wrote:
>
>> Andrey,
>>
>> I am not sure I understand. According to EntryProcessor API [1] you can
>> chose to return nothing.
>>
>> Also, to my knowledge, you can still do parallel reads while executing
>> the EntryProcessor. Perhaps other community members can elaborate on this.
>>
>> [1] https://static.javadoc.io/javax.cache/cache-api/1.0.0/in
>> dex.html?javax/cache/processor/EntryProcessor.html
>>
>> D.
>>
>>
>> On Mon, Aug 28, 2017 at 8:29 PM, Andrey Kornev <andrewkornev@hotmail.com>
>> wrote:
>>
>>> Dmitriy,
>>>
>>>
>>> It's good to be back! 😃 Glad to find Ignite community as vibrant
>>> and thriving as ever!
>>>
>>> Speaking of invokeAll(), even if we ignore for a moment the overhead
>>> associated with locking/unlocking a cache entry prior to passing it to the
>>> EntryProcessor as well as the overhead associated with enlisting the
>>> touched entries in a transaction, the bigger problem with using
>>> invokeAll() for filtering is that EntryProcessor must return a value. I'm
>>> not aware of any way to make EntryProcessor drop the entry from the
>>> response. The only options is to use a null (or false) to indicate a
>>> filtered out entry. In my specific case, I'll end up sending back a whole
>>> bunch of nulls in the result map as I expect most of the keys to be
>>> rejected by the filter.
>>>
>>> Overall, invokeAll() is not what one would call *efficient* (the key
>>> word in my original question) way of filtering.
>>>
>>> Thanks!
>>> Andrey
>>>
>>> ------------------------------
>>> *From:* Dmitriy Setrakyan <dsetrakyan@apache.org>
>>> *Sent:* Saturday, August 26, 2017 8:37 AM
>>> *To:* user
>>>
>>> *Subject:* Re: Retrieving multiple keys with filtering
>>>
>>> Andrey,
>>>
>>> Good to hear from you. Long time no talk.
>>>
>>> I don't think invokeAll has only update semantics. You can definitely
>>> use it just to look at the keys and return a result. Also, as you
>>> mentioned, Ignite compute is a viable option as well.
>>>
>>> The reason that predicates were removed from the get methods is because
>>> the API was becoming unwary, and also because JCache does not require it.
>>>
>>> D.
>>>
>>> On Thu, Aug 24, 2017 at 10:50 AM, Andrey Kornev <
>>> andrewkornev@hotmail.com> wrote:
>>>
>>>> Well, I believe invokeAll() has "update" semantics and using it for
>>>> read-only filtering of cache entries is probably not going to be efficient
>>>> or even appropriate.
>>>>
>>>>
>>>> I'm afraid the only viable option I'm left with is to use Ignite's
>>>> Compute feature:
>>>>
>>>> - on the sender, group the keys by affinity.
>>>>
>>>> - send each group along with the filter predicate to their
>>>> affinity nodes using IgniteCompute.
>>>>
>>>> - on each node, use getAll() to fetch the local keys and apply the
>>>> filter.
>>>>
>>>> - on the sender node, collect the results of the compute jobs into a
>>>> map.
>>>>
>>>>
>>>> It's unfortunate that Ignite dropped that original API. What used to be
>>>> a single API call is now a non-trivial algorithm and one have to worry
>>>> about things like what happens if the grid topology changes while the
>>>> compute jobs are executing, etc.
>>>>
>>>> Can anyone think of any other less complex/more robust approach?
>>>>
>>>> Thanks
>>>> Andrey
>>>>
>>>> ------------------------------
>>>> *From:* slava.koptilin <slava.koptilin@gmail.com>
>>>> *Sent:* Thursday, August 24, 2017 9:03 AM
>>>> *To:* user@ignite.apache.org
>>>> *Subject:* Re: Retrieving multiple keys with filtering
>>>>
>>>> Hi Andrey,
>>>>
>>>> Yes, you are right. ScanQuery scans all entries.
>>>> Perhaps, IgniteCache#invokeAll(keys, cacheEntryProcessor) with custom
>>>> processor will work for you.
>>>> https://ignite.apache.org/releases/2.1.0/javadoc/org/apache/
>>>> ignite/IgniteCache.html#invokeAll(java.util.Set,%20org.apach
>>>> e.ignite.cache.CacheEntryProcessor,%20java.lang.Object...)
>>>>
>>>> Thanks!
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context: http://apache-ignite-users.705
>>>> 18.x6.nabble.com/Retrieving-multiple-keys-with-filtering-tp1
>>>> 6391p16400.html
>>>> Apache Ignite Users - Retrieving multiple keys with filtering
>>>> <http://apache-ignite-users.70518.x6.nabble.com/Retrieving-multiple-keys-with-filtering-tp16391p16400.html>
>>>> apache-ignite-users.70518.x6.nabble.com
>>>> Retrieving multiple keys with filtering. Hello, I have a list of cache
>>>> keys (up to a few hundred of them) and a filter predicate. I'd like to
>>>> efficiently retrieve only those values that pass the...
>>>>
>>>>
>>>> Sent from the Apache Ignite Users mailing list archive at Nabble.com.
>>>>
>>>
>>>
>>
>

Mime
View raw message