ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Valentin Kulichenko <valentin.kuliche...@gmail.com>
Subject Re: Transformers in SCAN queries
Date Thu, 11 Feb 2016 03:01:38 GMT
I changed the description in the ticket. If anyone has comments about the
implementation details, feel free to add them there.

-Val

On Wed, Feb 10, 2016 at 2:30 PM, Dmitriy Setrakyan <dsetrakyan@apache.org>
wrote:

> I like the new method on IgniteCache API:
>
> <R> QueryCursor<R> query(Query<E> qry, Transformer<E, R> transformer);
>
> I do agree that it makes little sense for the SqlFieldsQuery, but we can
> simply throw an unsupported exception here. For all other queries it does
> make sense.
>
> Val, can you put together all the thoughts expressed in this thread into a
> ticket?
>
> D.
>
> On Thu, Feb 4, 2016 at 4:14 PM, Andrey Gura <agura@gridgain.com> wrote:
>
> > Val,
> >
> > From my point of view, special query class that allows transforming is
> > confusing. Two points about it:
> >
> > 1. ScanTransformQuery will duplicate ScanQuery code with all drawbacks
> that
> > we have. Moreover, any fix for ScanQuery should be repeated for
> > ScanTransformQuery. It will lead to bugs. DRY.
> >
> > 2. If some users want to do transformations for SqlQuery we will
> introduce
> > SqlTransformQuery. Right? At this point see previous item.
> >
> > Transformation logic is some kind of strategy. IMHO, the most convenient
> > API should get transformation logic as some function provided by user.
> >
> > On Thu, Feb 4, 2016 at 11:05 PM, Valentin Kulichenko <
> > valentin.kulichenko@gmail.com> wrote:
> >
> > > Agree with Sergi. Mixing SQL with Java code transformers is confusing.
> In
> > > rare case when it's really required, user can implement a custom
> > function.
> > >
> > > I copy-pasted the API to the ticket [1]. Please provide any additional
> > > comments there.
> > >
> > > [1] https://issues.apache.org/jira/browse/IGNITE-2546
> > >
> > > -Val
> > >
> > > On Thu, Feb 4, 2016 at 10:08 AM, Andrey Gura <agura@gridgain.com>
> wrote:
> > >
> > > > Sergi,
> > > >
> > > >
> > > > > What you are going to transform remotely here?
> > > >
> > > >
> > > > I'm not going. I believe :)
> > > >
> > > > Just hypothetical use case: You have one SqlFieldsQuery but different
> > > > requirements for returned values. For one case you have to return
> some
> > > > string fields as is and for another case you have to trim string to
> 32
> > > > characters. Of course you still can trim strings locally but
> > transformers
> > > > allow you do it remotely.
> > > >
> > > > Anyway, this solution can be usefull for rest query types.
> > > >
> > > > On Thu, Feb 4, 2016 at 8:54 PM, Sergi Vladykin <
> > sergi.vladykin@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > The whole point of Transformer is to do remote transform, how will
> > this
> > > > > work with SqlFieldsQuery? What you are going to transform remotely
> > > here?
> > > > I
> > > > > believe all the transformations must happen at SQL level here.
> > > > >
> > > > > Sergi
> > > > >
> > > > >
> > > > >
> > > > > 2016-02-04 20:10 GMT+03:00 Andrey Gura <agura@gridgain.com>:
> > > > >
> > > > > > SqlQuery, TextQuery and SpiQuery are similar to ScanQuery because
> > all
> > > > of
> > > > > > them also defined as Query<Cache.Entry<>>.
> > > > > >
> > > > > > It can be usefull to have one query SqlQuery that can return
> > > different
> > > > > > result that will be produced from cache entry by transformer.
> > > > > >
> > > > > > Actually only SqlFieldsQuery has different definition. So
> > > transformers
> > > > > can
> > > > > > be applied to any type of query (including SqlFieldsQuery, I
> > > believe).
> > > > > >
> > > > > > On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin <
> > > > sergi.vladykin@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I don't like the idea of having additional method
> *query(Query<E>
> > > > qry,
> > > > > > > Transformer<E, R> transfomer); *because I don't see
how these
> > > > > > transformers
> > > > > > > will work for example with SQL, but this API makes you
think
> that
> > > > > > > transformers are supported for all the query types.
> > > > > > >
> > > > > > > Sergi
> > > > > > >
> > > > > > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <agura@gridgain.com>:
> > > > > > >
> > > > > > > > Val,
> > > > > > > >
> > > > > > > > can we introduce new method into IgnteCache API?
> > > > > > > >
> > > > > > > > Now we have method: public <R> QueryCursor<R>
query(Query<R>
> > > qry);
> > > > > > > >
> > > > > > > > New method will be something like this: <R>
QueryCursor<R>
> > > > > > query(Query<E>
> > > > > > > > qry, Transformer<E, R> transfomer);
> > > > > > > >
> > > > > > > > It allows provide transformers for all query types
and
> chnages
> > > will
> > > > > be
> > > > > > > > related only with query cursor functionality.
> > > > > > > >
> > > > > > > > Will it work?
> > > > > > > >
> > > > > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev <
> > > > > > andrewkornev@hotmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Another perhaps bigger problem with running queries
> > (including
> > > > scan
> > > > > > > > > queries) using closures was discussed at length
on the @dev
> > not
> > > > so
> > > > > > long
> > > > > > > > > ago. It has to do with partitions migration due
to cluster
> > > > topology
> > > > > > > > changes
> > > > > > > > > which may result in the query returning incomplete
result.
> > And
> > > > > while
> > > > > > it
> > > > > > > > is
> > > > > > > > > possible to solve this problem for the scan queries
by
> using
> > > some
> > > > > > > clever
> > > > > > > > > tricks, all bets are off with the SQL queries.Andrey
> > > > > > > > >     _____________________________
> > > > > > > > > From: Valentin Kulichenko <valentin.kulichenko@gmail.com>
> > > > > > > > > Sent: Thursday, February 4, 2016 6:29 AM
> > > > > > > > > Subject: Re: Transformers in SCAN queries
> > > > > > > > > To:  <dev@ignite.apache.org>
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >                    Dmitry,
> > > > > > > > >
> > > > > > > > >  The main difference in my view is that you lose
pagination
> > > when
> > > > > > > sending
> > > > > > > > >  results from servers to client. What if one
wants to
> iterate
> > > > > through
> > > > > > > all
> > > > > > > > >  entries in cache?
> > > > > > > > >
> > > > > > > > >  On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy Setrakyan
<
> > > > > > > > dsetrakyan@apache.org>
> > > > > > > > >  wrote:
> > > > > > > > >
> > > > > > > > >  > Valentin,
> > > > > > > > >  >
> > > > > > > > >  > Wouldn’t the same effect be achieved
by broadcasting a
> > > closure
> > > > > to
> > > > > > > the
> > > > > > > > >  > cluster and executing scan-query on every
node locally?
> > > > > > > > >  >
> > > > > > > > >  > D.
> > > > > > > > >  >
> > > > > > > > >  > On Wed, Feb 3, 2016 at 9:17 PM, Valentin
Kulichenko <
> > > > > > > > >  > valentin.kulichenko@gmail.com> wrote:
> > > > > > > > >  >
> > > > > > > > >  > > Igniters,
> > > > > > > > >  > >
> > > > > > > > >  > > I keep getting requests from our users
to add optional
> > > > > > > transformers
> > > > > > > > to
> > > > > > > > >  > SCAN
> > > > > > > > >  > > queries. This will allow to iterate
through cache, but
> > do
> > > > not
> > > > > > > > transfer
> > > > > > > > >  > > whole key-value pairs across networks
(e.g., get only
> > > keys).
> > > > > The
> > > > > > > > > feature
> > > > > > > > >  > > looks useful and I created a ticket
[1].
> > > > > > > > >  > >
> > > > > > > > >  > > I am struggling with the design now.
The problem is
> > that I
> > > > > > wanted
> > > > > > > to
> > > > > > > > >  > extend
> > > > > > > > >  > > existing ScanQuery object for this,
but this seems to
> be
> > > > > > > impossible
> > > > > > > > >  > because
> > > > > > > > >  > > it already extends Query<Cache.Entry<K,
V>> and thus
> can
> > > > > iterate
> > > > > > > > only
> > > > > > > > >  > > through entries.
> > > > > > > > >  > >
> > > > > > > > >  > > The only option I see now is to create
a separate
> query
> > > > type,
> > > > > > > > > copy-paste
> > > > > > > > >  > > everything from ScanQuery and add
*mandatory*
> > transformer.
> > > > > > > Something
> > > > > > > > > like
> > > > > > > > >  > > this:
> > > > > > > > >  > >
> > > > > > > > >  > > ScanTransformQuery<K, V, R>
extends Query<R> {
> > > > > > > > >  > >     IgniteBiPredicate<K, V>
filter;
> > > > > > > > >  > >     IgniteClosure<Cache.Entry<K,
V>, R> transformer;
> > > > > > > > >  > >     int part;
> > > > > > > > >  > >     ...
> > > > > > > > >  > > }
> > > > > > > > >  > >
> > > > > > > > >  > > Thoughts? Does anyone has other ideas?
> > > > > > > > >  > >
> > > > > > > > >  > > [1]
> > https://issues.apache.org/jira/browse/IGNITE-2546
> > > > > > > > >  > >
> > > > > > > > >  > > -Val
> > > > > > > > >  > >
> > > > > > > > >  >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Andrey Gura
> > > > > > > > GridGain Systems, Inc.
> > > > > > > > www.gridgain.com
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Andrey Gura
> > > > > > GridGain Systems, Inc.
> > > > > > www.gridgain.com
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Andrey Gura
> > > > GridGain Systems, Inc.
> > > > www.gridgain.com
> > > >
> > >
> >
> >
> >
> > --
> > Andrey Gura
> > GridGain Systems, Inc.
> > www.gridgain.com
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message