ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitriy Setrakyan <dsetrak...@apache.org>
Subject Re: POJO store usability issues
Date Tue, 03 May 2016 16:10:39 GMT
Alexey,

What you are saying should already be possible by sending compute closures
to nodes and calling localLoadCache(“myCustomSqlStatement”).

Isn’t it just a case of providing proper example and documentation?

D.

On Tue, May 3, 2016 at 7:22 AM, Alexey Kuznetsov <akuznetsov@gridgain.com>
wrote:

> I totally agree with Vladimir.
>
> From JdbcPojo store side we could introduce support of some kind load
> descriptor
>  that will contains SQL to execute and node filter.
>
> On each node store will check node filter and execute SQL if node match the
> filter.
>
> This will solve first problem - "do not load full database on each node" .
>
> As for second problem - "not ignore non-primary non-backup entries" - I
> think this should be solved on a cache level,
>  because store does not know anything about primary / backup.
>
> Thoughts?
>
>
> On Wed, Apr 27, 2016 at 9:27 PM, Vladimir Ozerov <vozerov@gridgain.com>
> wrote:
>
> > Igniters,
> >
> > We receive more and more questions about the same problem: "I have big a
> > database. How should I load it to Ignite?"
> >
> > Obviously, users try to use POJO store as a most convenient approach, but
> > it cannot handle this case properly.
> > 1) If user invoke *IgniteCache.loadCache()*, then the same request - and
> > usually this is full table scan - will be invoked on every node leading
> to
> > very poor performance. For instance, we have a report of a load of 47M
> > entries to cache on 16 nodes which took ... 8 hours!!!
> > 2) If user invoke IgniteCache.localLoadCache(), then our internal cache
> > logic will filter out non-primary and non-backup entries. So this
> approach
> > doesn't work either.
> > 3) User could try using *IgniteDataStreamer*, but in this case he had to
> > deal with all JDBC-related stuff on his own - not convenient.
> > 4) Another approach I heard several times - "user should have an
> attribute
> > for affinity in the table ...". And the idea that this way user will be
> > able to divide the whole data set into several disjoint sets with
> specific
> > affinity. Doesn't work. Consider the user with some legacy database - the
> > most common use case. How is he going to work with affinity?
> >
> > Bottom line: Ignite has *no convenient way *to load millions of entries
> > from a database.
> >
> > We need to start thinking of possible solutions. Several ideas from my
> > side:
> >
> > 1) POJO store must be much more flexible. We should be able to pass
> > different queries to different nodes when calling "loadCache".
> >
> > 2) Cache store could have additional mode when it will not ignore
> > non-primary non-backup entries, but rather *distribute *it to other
> nodes.
> > E.g. with help of data streamer.
> >
> > Thoughts?
> >
> > Vladimir.
> >
>
>
>
> --
> Alexey Kuznetsov
> GridGain Systems
> www.gridgain.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message