ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Kuznetsov <akuznet...@gridgain.com>
Subject Re: POJO store usability issues
Date Wed, 04 May 2016 16:12:38 GMT
Dmitriy,

May be, but sending closures is seems as workaround for me.
This will require from user to write quite a lot of code.
Just call of loadCache(...some descriptors...) will be much more user
friendly.


Also I'm not sure what will happen with key loaded from DB if it will be
not affinity key for node where  localLoadCache will be executed.
Will it be ignored or put into cache?

Semen, Alexey Goncharuk - could you tell what will happen with such keys
inside of localLoadCache ?

On Tue, May 3, 2016 at 11:10 PM, Dmitriy Setrakyan <dsetrakyan@apache.org>
wrote:

> Alexey,
>
> What you are saying should already be possible by sending compute closures
> to nodes and calling localLoadCache(“myCustomSqlStatement”).
>
> Isn’t it just a case of providing proper example and documentation?
>
> D.
>
> On Tue, May 3, 2016 at 7:22 AM, Alexey Kuznetsov <akuznetsov@gridgain.com>
> wrote:
>
> > I totally agree with Vladimir.
> >
> > From JdbcPojo store side we could introduce support of some kind load
> > descriptor
> >  that will contains SQL to execute and node filter.
> >
> > On each node store will check node filter and execute SQL if node match
> the
> > filter.
> >
> > This will solve first problem - "do not load full database on each node"
> .
> >
> > As for second problem - "not ignore non-primary non-backup entries" - I
> > think this should be solved on a cache level,
> >  because store does not know anything about primary / backup.
> >
> > Thoughts?
> >
> >
> > On Wed, Apr 27, 2016 at 9:27 PM, Vladimir Ozerov <vozerov@gridgain.com>
> > wrote:
> >
> > > Igniters,
> > >
> > > We receive more and more questions about the same problem: "I have big
> a
> > > database. How should I load it to Ignite?"
> > >
> > > Obviously, users try to use POJO store as a most convenient approach,
> but
> > > it cannot handle this case properly.
> > > 1) If user invoke *IgniteCache.loadCache()*, then the same request -
> and
> > > usually this is full table scan - will be invoked on every node leading
> > to
> > > very poor performance. For instance, we have a report of a load of 47M
> > > entries to cache on 16 nodes which took ... 8 hours!!!
> > > 2) If user invoke IgniteCache.localLoadCache(), then our internal cache
> > > logic will filter out non-primary and non-backup entries. So this
> > approach
> > > doesn't work either.
> > > 3) User could try using *IgniteDataStreamer*, but in this case he had
> to
> > > deal with all JDBC-related stuff on his own - not convenient.
> > > 4) Another approach I heard several times - "user should have an
> > attribute
> > > for affinity in the table ...". And the idea that this way user will be
> > > able to divide the whole data set into several disjoint sets with
> > specific
> > > affinity. Doesn't work. Consider the user with some legacy database -
> the
> > > most common use case. How is he going to work with affinity?
> > >
> > > Bottom line: Ignite has *no convenient way *to load millions of entries
> > > from a database.
> > >
> > > We need to start thinking of possible solutions. Several ideas from my
> > > side:
> > >
> > > 1) POJO store must be much more flexible. We should be able to pass
> > > different queries to different nodes when calling "loadCache".
> > >
> > > 2) Cache store could have additional mode when it will not ignore
> > > non-primary non-backup entries, but rather *distribute *it to other
> > nodes.
> > > E.g. with help of data streamer.
> > >
> > > Thoughts?
> > >
> > > Vladimir.
> > >
> >
> >
> >
> > --
> > Alexey Kuznetsov
> > GridGain Systems
> > www.gridgain.com
> >
>



-- 
Alexey Kuznetsov
GridGain Systems
www.gridgain.com

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message