ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: Transformers in SCAN queries
Date Thu, 11 Feb 2016 17:44:36 GMT
On Thu, Feb 11, 2016 at 6:18 AM, Andrey Gura <agura@gridgain.com> wrote:

> Despite of trasformer makes little sense for the SqlFieldsQuery I don't see
> any reason for throwing of UnsupportedOperationException.
> I think it is stange limitation from user standpoint.
>

I think either supporting it or not supporting looks strange for
SqlFieldsQuery. I suggested to throw the exception only to simplify the
development - simply one less thing to worry about.


>
> On Thu, Feb 11, 2016 at 6:01 AM, Valentin Kulichenko <
> valentin.kulichenko@gmail.com> wrote:
>
> > I changed the description in the ticket. If anyone has comments about the
> > implementation details, feel free to add them there.
> >
> > -Val
> >
> > On Wed, Feb 10, 2016 at 2:30 PM, Dmitriy Setrakyan <
> dsetrakyan@apache.org>
> > wrote:
> >
> > > I like the new method on IgniteCache API:
> > >
> > > <R> QueryCursor<R> query(Query<E> qry, Transformer<E,
R> transformer);
> > >
> > > I do agree that it makes little sense for the SqlFieldsQuery, but we
> can
> > > simply throw an unsupported exception here. For all other queries it
> does
> > > make sense.
> > >
> > > Val, can you put together all the thoughts expressed in this thread
> into
> > a
> > > ticket?
> > >
> > > D.
> > >
> > > On Thu, Feb 4, 2016 at 4:14 PM, Andrey Gura <agura@gridgain.com>
> wrote:
> > >
> > > > Val,
> > > >
> > > > From my point of view, special query class that allows transforming
> is
> > > > confusing. Two points about it:
> > > >
> > > > 1. ScanTransformQuery will duplicate ScanQuery code with all
> drawbacks
> > > that
> > > > we have. Moreover, any fix for ScanQuery should be repeated for
> > > > ScanTransformQuery. It will lead to bugs. DRY.
> > > >
> > > > 2. If some users want to do transformations for SqlQuery we will
> > > introduce
> > > > SqlTransformQuery. Right? At this point see previous item.
> > > >
> > > > Transformation logic is some kind of strategy. IMHO, the most
> > convenient
> > > > API should get transformation logic as some function provided by
> user.
> > > >
> > > > On Thu, Feb 4, 2016 at 11:05 PM, Valentin Kulichenko <
> > > > valentin.kulichenko@gmail.com> wrote:
> > > >
> > > > > Agree with Sergi. Mixing SQL with Java code transformers is
> > confusing.
> > > In
> > > > > rare case when it's really required, user can implement a custom
> > > > function.
> > > > >
> > > > > I copy-pasted the API to the ticket [1]. Please provide any
> > additional
> > > > > comments there.
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-2546
> > > > >
> > > > > -Val
> > > > >
> > > > > On Thu, Feb 4, 2016 at 10:08 AM, Andrey Gura <agura@gridgain.com>
> > > wrote:
> > > > >
> > > > > > Sergi,
> > > > > >
> > > > > >
> > > > > > > What you are going to transform remotely here?
> > > > > >
> > > > > >
> > > > > > I'm not going. I believe :)
> > > > > >
> > > > > > Just hypothetical use case: You have one SqlFieldsQuery but
> > different
> > > > > > requirements for returned values. For one case you have to return
> > > some
> > > > > > string fields as is and for another case you have to trim string
> to
> > > 32
> > > > > > characters. Of course you still can trim strings locally but
> > > > transformers
> > > > > > allow you do it remotely.
> > > > > >
> > > > > > Anyway, this solution can be usefull for rest query types.
> > > > > >
> > > > > > On Thu, Feb 4, 2016 at 8:54 PM, Sergi Vladykin <
> > > > sergi.vladykin@gmail.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > The whole point of Transformer is to do remote transform,
how
> > will
> > > > this
> > > > > > > work with SqlFieldsQuery? What you are going to transform
> > remotely
> > > > > here?
> > > > > > I
> > > > > > > believe all the transformations must happen at SQL level
here.
> > > > > > >
> > > > > > > Sergi
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > 2016-02-04 20:10 GMT+03:00 Andrey Gura <agura@gridgain.com>:
> > > > > > >
> > > > > > > > SqlQuery, TextQuery and SpiQuery are similar to ScanQuery
> > because
> > > > all
> > > > > > of
> > > > > > > > them also defined as Query<Cache.Entry<>>.
> > > > > > > >
> > > > > > > > It can be usefull to have one query SqlQuery that
can return
> > > > > different
> > > > > > > > result that will be produced from cache entry by transformer.
> > > > > > > >
> > > > > > > > Actually only SqlFieldsQuery has different definition.
So
> > > > > transformers
> > > > > > > can
> > > > > > > > be applied to any type of query (including SqlFieldsQuery,
I
> > > > > believe).
> > > > > > > >
> > > > > > > > On Thu, Feb 4, 2016 at 7:42 PM, Sergi Vladykin <
> > > > > > sergi.vladykin@gmail.com
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I don't like the idea of having additional method
> > > *query(Query<E>
> > > > > > qry,
> > > > > > > > > Transformer<E, R> transfomer); *because
I don't see how
> these
> > > > > > > > transformers
> > > > > > > > > will work for example with SQL, but this API
makes you
> think
> > > that
> > > > > > > > > transformers are supported for all the query
types.
> > > > > > > > >
> > > > > > > > > Sergi
> > > > > > > > >
> > > > > > > > > 2016-02-04 16:46 GMT+03:00 Andrey Gura <agura@gridgain.com
> >:
> > > > > > > > >
> > > > > > > > > > Val,
> > > > > > > > > >
> > > > > > > > > > can we introduce new method into IgnteCache
API?
> > > > > > > > > >
> > > > > > > > > > Now we have method: public <R> QueryCursor<R>
> > query(Query<R>
> > > > > qry);
> > > > > > > > > >
> > > > > > > > > > New method will be something like this:
<R>
> QueryCursor<R>
> > > > > > > > query(Query<E>
> > > > > > > > > > qry, Transformer<E, R> transfomer);
> > > > > > > > > >
> > > > > > > > > > It allows provide transformers for all query
types and
> > > chnages
> > > > > will
> > > > > > > be
> > > > > > > > > > related only with query cursor functionality.
> > > > > > > > > >
> > > > > > > > > > Will it work?
> > > > > > > > > >
> > > > > > > > > > On Thu, Feb 4, 2016 at 11:13 AM, Andrey
Kornev <
> > > > > > > > andrewkornev@hotmail.com
> > > > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Another perhaps bigger problem with
running queries
> > > > (including
> > > > > > scan
> > > > > > > > > > > queries) using closures was discussed
at length on the
> > @dev
> > > > not
> > > > > > so
> > > > > > > > long
> > > > > > > > > > > ago. It has to do with partitions migration
due to
> > cluster
> > > > > > topology
> > > > > > > > > > changes
> > > > > > > > > > > which may result in the query returning
incomplete
> > result.
> > > > And
> > > > > > > while
> > > > > > > > it
> > > > > > > > > > is
> > > > > > > > > > > possible to solve this problem for
the scan queries by
> > > using
> > > > > some
> > > > > > > > > clever
> > > > > > > > > > > tricks, all bets are off with the SQL
queries.Andrey
> > > > > > > > > > >     _____________________________
> > > > > > > > > > > From: Valentin Kulichenko <
> valentin.kulichenko@gmail.com
> > >
> > > > > > > > > > > Sent: Thursday, February 4, 2016 6:29
AM
> > > > > > > > > > > Subject: Re: Transformers in SCAN queries
> > > > > > > > > > > To:  <dev@ignite.apache.org>
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >                    Dmitry,
> > > > > > > > > > >
> > > > > > > > > > >  The main difference in my view is
that you lose
> > pagination
> > > > > when
> > > > > > > > > sending
> > > > > > > > > > >  results from servers to client. What
if one wants to
> > > iterate
> > > > > > > through
> > > > > > > > > all
> > > > > > > > > > >  entries in cache?
> > > > > > > > > > >
> > > > > > > > > > >  On Wed, Feb 3, 2016 at 9:47 PM, Dmitriy
Setrakyan <
> > > > > > > > > > dsetrakyan@apache.org>
> > > > > > > > > > >  wrote:
> > > > > > > > > > >
> > > > > > > > > > >  > Valentin,
> > > > > > > > > > >  >
> > > > > > > > > > >  > Wouldn’t the same effect be
achieved by
> broadcasting a
> > > > > closure
> > > > > > > to
> > > > > > > > > the
> > > > > > > > > > >  > cluster and executing scan-query
on every node
> > locally?
> > > > > > > > > > >  >
> > > > > > > > > > >  > D.
> > > > > > > > > > >  >
> > > > > > > > > > >  > On Wed, Feb 3, 2016 at 9:17 PM,
Valentin Kulichenko
> <
> > > > > > > > > > >  > valentin.kulichenko@gmail.com>
wrote:
> > > > > > > > > > >  >
> > > > > > > > > > >  > > Igniters,
> > > > > > > > > > >  > >
> > > > > > > > > > >  > > I keep getting requests
from our users to add
> > optional
> > > > > > > > > transformers
> > > > > > > > > > to
> > > > > > > > > > >  > SCAN
> > > > > > > > > > >  > > queries. This will allow
to iterate through cache,
> > but
> > > > do
> > > > > > not
> > > > > > > > > > transfer
> > > > > > > > > > >  > > whole key-value pairs across
networks (e.g., get
> > only
> > > > > keys).
> > > > > > > The
> > > > > > > > > > > feature
> > > > > > > > > > >  > > looks useful and I created
a ticket [1].
> > > > > > > > > > >  > >
> > > > > > > > > > >  > > I am struggling with the
design now. The problem
> is
> > > > that I
> > > > > > > > wanted
> > > > > > > > > to
> > > > > > > > > > >  > extend
> > > > > > > > > > >  > > existing ScanQuery object
for this, but this seems
> > to
> > > be
> > > > > > > > > impossible
> > > > > > > > > > >  > because
> > > > > > > > > > >  > > it already extends Query<Cache.Entry<K,
V>> and
> thus
> > > can
> > > > > > > iterate
> > > > > > > > > > only
> > > > > > > > > > >  > > through entries.
> > > > > > > > > > >  > >
> > > > > > > > > > >  > > The only option I see now
is to create a separate
> > > query
> > > > > > type,
> > > > > > > > > > > copy-paste
> > > > > > > > > > >  > > everything from ScanQuery
and add *mandatory*
> > > > transformer.
> > > > > > > > > Something
> > > > > > > > > > > like
> > > > > > > > > > >  > > this:
> > > > > > > > > > >  > >
> > > > > > > > > > >  > > ScanTransformQuery<K,
V, R> extends Query<R> {
> > > > > > > > > > >  > >     IgniteBiPredicate<K,
V> filter;
> > > > > > > > > > >  > >     IgniteClosure<Cache.Entry<K,
V>, R>
> transformer;
> > > > > > > > > > >  > >     int part;
> > > > > > > > > > >  > >     ...
> > > > > > > > > > >  > > }
> > > > > > > > > > >  > >
> > > > > > > > > > >  > > Thoughts? Does anyone has
other ideas?
> > > > > > > > > > >  > >
> > > > > > > > > > >  > > [1]
> > > > https://issues.apache.org/jira/browse/IGNITE-2546
> > > > > > > > > > >  > >
> > > > > > > > > > >  > > -Val
> > > > > > > > > > >  > >
> > > > > > > > > > >  >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > > Andrey Gura
> > > > > > > > > > GridGain Systems, Inc.
> > > > > > > > > > www.gridgain.com
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > > Andrey Gura
> > > > > > > > GridGain Systems, Inc.
> > > > > > > > www.gridgain.com
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Andrey Gura
> > > > > > GridGain Systems, Inc.
> > > > > > www.gridgain.com
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Andrey Gura
> > > > GridGain Systems, Inc.
> > > > www.gridgain.com
> > > >
> > >
> >
>
>
>
> --
> Andrey Gura
> GridGain Systems, Inc.
> www.gridgain.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message