ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: IgniteCache.invoke on ALL keys
Date Thu, 02 Jun 2016 16:06:26 GMT
I responded in the ticket:
https://issues.apache.org/jira/browse/IGNITE-3222

On Thu, Jun 2, 2016 at 2:10 AM, Pavel Tupitsyn <ptupitsyn@gridgain.com>
wrote:

> Dmitriy, affinityCall/Run over a partition is useful, I agree.
>
> But dealing with partitions manually is also error-prone and not obvious at
> all.
> Invoke over all cache entries is a simple common task, why not provide a
> simple method of doing it?
>
> What kind of memory issues do you expect?
> Returning EntryProcessorResult for each entry on a huge cache is an obvious
> problem,
> same as calling getAll on QueryCursor, for example. We can't protect users
> from such things.
>
> Pavel.
>
> On Wed, Jun 1, 2016 at 3:18 AM, Dmitriy Setrakyan <dsetrakyan@apache.org>
> wrote:
>
> > Pavel,
> >
> > I actually believe that such method will be error-prone and will cause
> all
> > sorts of memory issues for users trying to execute this method over large
> > caches.
> >
> > What we need instead is an affinityCall/Run method over a partition, not
> a
> > key. Why not provide this method instead?
> >
> > Added my comment to the ticket.
> >
> > D.
> >
> > On Tue, May 31, 2016 at 6:18 AM, Pavel Tupitsyn <ptupitsyn@gridgain.com>
> > wrote:
> >
> > > Ok then, looks like there are no obstacles or objections, I've created
> a
> > > JIRA ticket:
> > > https://issues.apache.org/jira/browse/IGNITE-3222
> > >
> > > Thanks,
> > > Pavel.
> > >
> > > On Tue, May 31, 2016 at 3:44 PM, Denis Magda <dmagda@gridgain.com>
> > wrote:
> > >
> > > > Pavel,
> > > >
> > > > > This seems to be a common task, why don't we implement invokeAll
> as I
> > > > > suggested above?
> > > >
> > > > I would implement such a method using the approach shown in the
> > example.
> > > > In my understanding it would be the most efficient way. Data locality
> > > will
> > > > longer be not an issue when IGNITE-2310 is implemented.
> > > >
> > > > —
> > > > Denis
> > > >
> > > > > On May 31, 2016, at 1:16 PM, Pavel Tupitsyn <
> ptupitsyn@gridgain.com>
> > > > wrote:
> > > > >
> > > > > Denis, thank you, this may work, but:
> > > > > * it is too complicated
> > > > > * it does not guarantee locality
> > > > >
> > > > > This seems to be a common task, why don't we implement invokeAll
> as I
> > > > > suggested above?
> > > > >
> > > > > On Tue, May 31, 2016 at 1:05 PM, Denis Magda <dmagda@gridgain.com>
> > > > wrote:
> > > > >
> > > > >> Pavel,
> > > > >>
> > > > >> Here is an example on how to execute scan queries on partitions’
> > > owners
> > > > >> and perform some operation for every entry
> > > > >>
> > > > >>
> > > >
> > >
> >
> https://github.com/gridgain/gridgain-advanced-examples/blob/master/src/main/java/org/gridgain/examples/datagrid/query/ScanQueryExample.java
> > > > >> <
> > > > >>
> > > >
> > >
> >
> https://github.com/gridgain/gridgain-advanced-examples/blob/master/src/main/java/org/gridgain/examples/datagrid/query/ScanQueryExample.java
> > > > >>>
> > > > >>
> > > > >> When this feature is implemented [1] it will be possible to
> postpone
> > > > >> partitions movement until such an operation is in progress.
> > Presently
> > > if
> > > > >> the rebalancing happens the data will be retrieved from a remote
> > node
> > > > (new
> > > > >> partition owner), however the result will be consistent in any
> case.
> > > > >>
> > > > >> [1] https://issues.apache.org/jira/browse/IGNITE-2310 <
> > > > >> https://issues.apache.org/jira/browse/IGNITE-2310>
> > > > >>
> > > > >> —
> > > > >> Denis
> > > > >>
> > > > >>> On May 31, 2016, at 7:42 AM, Yakov Zhdanov <
> yzhdanov@gridgain.com>
> > > > >> wrote:
> > > > >>>
> > > > >>> Vova, even though it operates on single key we will need
to "pin"
> > > > >> partition
> > > > >>> so key does not go to another node.
> > > > >>>
> > > > >>> Pavel, you can also send closures to all primary nodes to
do
> local
> > > scan
> > > > >>> query for each partition. This way you will go over each
entry.
> > > > >>>
> > > > >>> Thanks!
> > > > >>> --
> > > > >>> Yakov Zhdanov, Director R&D
> > > > >>> *GridGain Systems*
> > > > >>> www.gridgain.com
> > > > >>>
> > > > >>> 2016-05-30 16:05 GMT-04:00 Vladimir Ozerov <vozerov@gridgain.com
> >:
> > > > >>>
> > > > >>>> Affinity run/call operate on a single key AFAIK.
> > > > >>>>
> > > > >>>> On Mon, May 30, 2016 at 10:55 PM, Dmitriy Setrakyan <
> > > > >> dsetrakyan@apache.org
> > > > >>>>>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>>> Actually I have seen a ticket to block moving partitions
if
> > > > >> affinityCall
> > > > >>>> or
> > > > >>>>> affinityRun are called. I think once these tickets
are
> > implemented,
> > > > the
> > > > >>>>> process will become reliable, no?
> > > > >>>>>
> > > > >>>>> D.
> > > > >>>>>
> > > > >>>>> On Mon, May 30, 2016 at 9:13 AM, Pavel Tupitsyn <
> > > > >> ptupitsyn@gridgain.com>
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>> Dmitriy, as I understand, there is no reliable
way to do that
> if
> > > > >>>>>> rebalancing happens.
> > > > >>>>>>
> > > > >>>>>> On Mon, May 30, 2016 at 6:50 PM, Dmitriy Setrakyan
<
> > > > >>>>> dsetrakyan@apache.org>
> > > > >>>>>> wrote:
> > > > >>>>>>
> > > > >>>>>>> I think we do support this use case. Why
not send a
> computation
> > > to
> > > > a
> > > > >>>>>> server
> > > > >>>>>>> and then perform the iteration through the
cache entries
> > locally
> > > on
> > > > >>>>> that
> > > > >>>>>>> server?
> > > > >>>>>>>
> > > > >>>>>>> On Mon, May 30, 2016 at 4:44 AM, Pavel Tupitsyn
<
> > > > >>>>> ptupitsyn@gridgain.com>
> > > > >>>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>> Igniters,
> > > > >>>>>>>>
> > > > >>>>>>>> Looks like we do not have an efficient
way to perform an
> > action
> > > on
> > > > >>>>>> EVERY
> > > > >>>>>>>> cache entry.
> > > > >>>>>>>>
> > > > >>>>>>>> Let's say I want to remove all entries
that match a
> predicate.
> > > > >>>>>>>> My only option is to retrieve these entries
via Scan or SQL
> > > query,
> > > > >>>>> and
> > > > >>>>>>> then
> > > > >>>>>>>> call removeAll.
> > > > >>>>>>>> This involves a lot of unnecessary network
trips (send keys
> to
> > > > >>>> caller
> > > > >>>>>>> node,
> > > > >>>>>>>> send them back to primary nodes).
> > > > >>>>>>>>
> > > > >>>>>>>> Would it be possible to implement a method
like
> > > > >>>>>>>> void IgniteCache.invokeAll(entryProcessor)
> > > > >>>>>>>> that invokes the processor on all entries
and does not
> return
> > > > >>>>> anything?
> > > > >>>>>>>> There could be more overloads that return
results or only
> > return
> > > > >>>>>> results
> > > > >>>>>>>> for changed entries.
> > > > >>>>>>>>
> > > > >>>>>>>> Thoughts?
> > > > >>>>>>>>
> > > > >>>>>>>> Pavel.
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>>
> > > > >>>>
> > > > >>
> > > > >>
> > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message