ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrey Gura <ag...@gridgain.com>
Subject Re: Transformers in SCAN queries
Date Thu, 11 Feb 2016 14:18:05 GMT
Despite of trasformer makes little sense for the SqlFieldsQuery I don't see
any reason for throwing of UnsupportedOperationException.
I think it is stange limitation from user standpoint.

On Thu, Feb 11, 2016 at 6:01 AM, Valentin Kulichenko <
valentin.kulichenko@gmail.com> wrote:

> I changed the description in the ticket. If anyone has comments about the
> implementation details, feel free to add them there.
>
> -Val
>
> On Wed, Feb 10, 2016 at 2:30 PM, Dmitriy Setrakyan <dsetrakyan@apache.org>
> wrote:
>
> > I like the new method on IgniteCache API:
> >
> > <R> QueryCursor<R> query(Query<E> qry, Transformer<E, R>
transformer);
> >
> > I do agree that it makes little sense for the SqlFieldsQuery, but we can
> > simply throw an unsupported exception here. For all other queries it does
> > make sense.
> >
> > Val, can you put together all the thoughts expressed in this thread into
> a
> > ticket?
> >
> > D.
> >
> > On Thu, Feb 4, 2016 at 4:14 PM, Andrey Gura <agura@gridgain.com> wrote:
> >
> > > Val,
> > >
> > > From my point of view, special query class that allows transforming is
> > > confusing. Two points about it:
> > >
> > > 1. ScanTransformQuery will duplicate ScanQuery code with all drawbacks
> > that
> > > we have. Moreover, any fix for ScanQuery should be repeated for
> > > ScanTransformQuery. It will lead to bugs. DRY.
> > >
> > > 2. If some users want to do transformations for SqlQuery we will
> > introduce
> > > SqlTransformQuery. Right? At this point see previous item.
> > >
> > > Transformation logic is some kind of strategy. IMHO, the most
> convenient
> > > API should get transformation logic as some function provided by user.
> > >
> > > On Thu, Feb 4, 2016 at 11:05 PM, Valentin Kulichenko <
> > > valentin.kulichenko@gmail.com> wrote:
> > >
> > > > Agree with Sergi. Mixing SQL with Java code transformers is
> confusing.
> > In
> > > > rare case when it's really required, user can implement a custom
> > > function.
> > > >
> > > > I copy-pasted the API to the ticket [1]. Please provide any
> additional
> > > > comments there.
> > > >
> > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546
> > > >
> > > > -Val
> > > >
> > > > On Thu, Feb 4, 2016 at 10:08 AM, Andrey Gura <agura@gridgain.com>
> > wrote:
> > > >
> > > > > Sergi,
> > > > >
> > > > >
> > > > > > What you are going to transform remotely here?
> > > > >
> > > > >
> > > > > I'm not going. I believe :)
> > > > >
> > > > > Just hypothetical use case: You have one SqlFieldsQuery but
> different
> > > > > requirements for returned values. For one case you have to return
> > some
> > > > > string fields as is and for another case you have to trim string
to
> > 32
> > > > > characters. Of course you still can trim strings locally but
> > > transformers
> > > > > allow you do it remotely.
> > > > >
> > > > > Anyway, this solution can be usefull for rest query types.
> > > > >
> > > > > On Thu, Feb 4, 2016 at 8:54 PM, Sergi Vladykin <
> > > sergi.vladykin@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > The whole point of Transformer is to do remote transform, how
> will
> > > this
> > > > > > work with SqlFieldsQuery? What you are going to transform
> remotely
> > > > here?
> > > > > I
> > > > > > believe all the transformations must happen at SQL level here.
> > > > > >
> > > > > > Sergi
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2016-02-04 20:10 GMT+03:00 Andrey Gura <agura@gridgain.com>:
> > > > > >
> > > > > > > SqlQuery, TextQuery and SpiQuery are similar to ScanQuery
> because
> > > all
> > > > > of
> > > > > > > them also defined as Query<Cache.Entry<>>.
> > > > > > >
> > > > > > > It can be usefull to have one query SqlQuery that can return
> > > > different
> > > > > > > result that will be produced from cache entry by transformer.
> > > > > > >
> > > > > > > Actually only SqlFieldsQuery has different definition.
So
> > > > transformers
> > > > > > can
> > > > > > > be applied to any type of query (including SqlFieldsQuery,
I
> > > > believe).
> > > > > > >
> > > > > > > On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin <
> > > > > sergi.vladykin@gmail.com
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I don't like the idea of having additional method
> > *query(Query<E>
> > > > > qry,
> > > > > > > > Transformer<E, R> transfomer); *because I don't
see how these
> > > > > > > transformers
> > > > > > > > will work for example with SQL, but this API makes
you think
> > that
> > > > > > > > transformers are supported for all the query types.
> > > > > > > >
> > > > > > > > Sergi
> > > > > > > >
> > > > > > > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <agura@gridgain.com>:
> > > > > > > >
> > > > > > > > > Val,
> > > > > > > > >
> > > > > > > > > can we introduce new method into IgnteCache API?
> > > > > > > > >
> > > > > > > > > Now we have method: public <R> QueryCursor<R>
> query(Query<R>
> > > > qry);
> > > > > > > > >
> > > > > > > > > New method will be something like this: <R>
QueryCursor<R>
> > > > > > > query(Query<E>
> > > > > > > > > qry, Transformer<E, R> transfomer);
> > > > > > > > >
> > > > > > > > > It allows provide transformers for all query
types and
> > chnages
> > > > will
> > > > > > be
> > > > > > > > > related only with query cursor functionality.
> > > > > > > > >
> > > > > > > > > Will it work?
> > > > > > > > >
> > > > > > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey Kornev
<
> > > > > > > andrewkornev@hotmail.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Another perhaps bigger problem with running
queries
> > > (including
> > > > > scan
> > > > > > > > > > queries) using closures was discussed at
length on the
> @dev
> > > not
> > > > > so
> > > > > > > long
> > > > > > > > > > ago. It has to do with partitions migration
due to
> cluster
> > > > > topology
> > > > > > > > > changes
> > > > > > > > > > which may result in the query returning
incomplete
> result.
> > > And
> > > > > > while
> > > > > > > it
> > > > > > > > > is
> > > > > > > > > > possible to solve this problem for the scan
queries by
> > using
> > > > some
> > > > > > > > clever
> > > > > > > > > > tricks, all bets are off with the SQL queries.Andrey
> > > > > > > > > >     _____________________________
> > > > > > > > > > From: Valentin Kulichenko <valentin.kulichenko@gmail.com
> >
> > > > > > > > > > Sent: Thursday, February 4, 2016 6:29 AM
> > > > > > > > > > Subject: Re: Transformers in SCAN queries
> > > > > > > > > > To:  <dev@ignite.apache.org>
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >                    Dmitry,
> > > > > > > > > >
> > > > > > > > > >  The main difference in my view is that
you lose
> pagination
> > > > when
> > > > > > > > sending
> > > > > > > > > >  results from servers to client. What if
one wants to
> > iterate
> > > > > > through
> > > > > > > > all
> > > > > > > > > >  entries in cache?
> > > > > > > > > >
> > > > > > > > > >  On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy
Setrakyan <
> > > > > > > > > dsetrakyan@apache.org>
> > > > > > > > > >  wrote:
> > > > > > > > > >
> > > > > > > > > >  > Valentin,
> > > > > > > > > >  >
> > > > > > > > > >  > Wouldn’t the same effect be achieved
by broadcasting a
> > > > closure
> > > > > > to
> > > > > > > > the
> > > > > > > > > >  > cluster and executing scan-query on
every node
> locally?
> > > > > > > > > >  >
> > > > > > > > > >  > D.
> > > > > > > > > >  >
> > > > > > > > > >  > On Wed, Feb 3, 2016 at 9:17 PM, Valentin
Kulichenko <
> > > > > > > > > >  > valentin.kulichenko@gmail.com>
wrote:
> > > > > > > > > >  >
> > > > > > > > > >  > > Igniters,
> > > > > > > > > >  > >
> > > > > > > > > >  > > I keep getting requests from
our users to add
> optional
> > > > > > > > transformers
> > > > > > > > > to
> > > > > > > > > >  > SCAN
> > > > > > > > > >  > > queries. This will allow to iterate
through cache,
> but
> > > do
> > > > > not
> > > > > > > > > transfer
> > > > > > > > > >  > > whole key-value pairs across
networks (e.g., get
> only
> > > > keys).
> > > > > > The
> > > > > > > > > > feature
> > > > > > > > > >  > > looks useful and I created a
ticket [1].
> > > > > > > > > >  > >
> > > > > > > > > >  > > I am struggling with the design
now. The problem is
> > > that I
> > > > > > > wanted
> > > > > > > > to
> > > > > > > > > >  > extend
> > > > > > > > > >  > > existing ScanQuery object for
this, but this seems
> to
> > be
> > > > > > > > impossible
> > > > > > > > > >  > because
> > > > > > > > > >  > > it already extends Query<Cache.Entry<K,
V>> and thus
> > can
> > > > > > iterate
> > > > > > > > > only
> > > > > > > > > >  > > through entries.
> > > > > > > > > >  > >
> > > > > > > > > >  > > The only option I see now is
to create a separate
> > query
> > > > > type,
> > > > > > > > > > copy-paste
> > > > > > > > > >  > > everything from ScanQuery and
add *mandatory*
> > > transformer.
> > > > > > > > Something
> > > > > > > > > > like
> > > > > > > > > >  > > this:
> > > > > > > > > >  > >
> > > > > > > > > >  > > ScanTransformQuery<K, V, R>
extends Query<R> {
> > > > > > > > > >  > >     IgniteBiPredicate<K, V>
filter;
> > > > > > > > > >  > >     IgniteClosure<Cache.Entry<K,
V>, R> transformer;
> > > > > > > > > >  > >     int part;
> > > > > > > > > >  > >     ...
> > > > > > > > > >  > > }
> > > > > > > > > >  > >
> > > > > > > > > >  > > Thoughts? Does anyone has other
ideas?
> > > > > > > > > >  > >
> > > > > > > > > >  > > [1]
> > > https://issues.apache.org/jira/browse/IGNITE-2546
> > > > > > > > > >  > >
> > > > > > > > > >  > > -Val
> > > > > > > > > >  > >
> > > > > > > > > >  >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Andrey Gura
> > > > > > > > > GridGain Systems, Inc.
> > > > > > > > > www.gridgain.com
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Andrey Gura
> > > > > > > GridGain Systems, Inc.
> > > > > > > www.gridgain.com
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Andrey Gura
> > > > > GridGain Systems, Inc.
> > > > > www.gridgain.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Andrey Gura
> > > GridGain Systems, Inc.
> > > www.gridgain.com
> > >
> >
>



-- 
Andrey Gura
GridGain Systems, Inc.
www.gridgain.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message