ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anghel Botos (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-4521) Allow org.apache.ignite.IgniteCache#loadCache to use also a producer/stream of custom queries
Date Wed, 04 Jan 2017 08:30:59 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-4521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Anghel Botos updated IGNITE-4521:
---------------------------------
    Description: 
As suggested in the comments from https://issues.apache.org/jira/browse/IGNITE-4164, we are
using {{org.apache.ignite.IgniteCache#loadCache}} with a large number of custom queries to
allow for the parallel load of the cache. The custom queries are created by our code in a
similar manner as in {{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}}
when called with no custom queries. The problem that we face is the fact that the generation
of these queries takes a while (on the order of minutes for our data set), time which could
already be spent loading data into the cache, once the first queries are available.

The suggestion for improvement is to change the signature of the {{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}}
and {{org.apache.ignite.IgniteCache#loadCache}} method to use a Producer/Stream for the custom
queries, and the workers in the thread pool doing the actual loading should just get queries
from this stream until it is exhausted/get from the producer until it returns null (or some
other indicator that there are no more queries to be produced). This would shorten the total
time required to load the cache, as it would make it more similar in behavior to the part
of {{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}} handling the case
without custom queries (where as soon as the boundaries of a range are retrieved from the
initial {{loadCacheSelRangeQry}}, the load of the range is automatically submitted to the
thread pool for execution) 

  was:As suggested in the comments from When using org.apache.ignite.IgniteCache#loadCache
with 


> Allow org.apache.ignite.IgniteCache#loadCache to use also a producer/stream of custom
queries
> ---------------------------------------------------------------------------------------------
>
>                 Key: IGNITE-4521
>                 URL: https://issues.apache.org/jira/browse/IGNITE-4521
>             Project: Ignite
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 1.8
>            Reporter: Anghel Botos
>
> As suggested in the comments from https://issues.apache.org/jira/browse/IGNITE-4164,
we are using {{org.apache.ignite.IgniteCache#loadCache}} with a large number of custom queries
to allow for the parallel load of the cache. The custom queries are created by our code in
a similar manner as in {{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}}
when called with no custom queries. The problem that we face is the fact that the generation
of these queries takes a while (on the order of minutes for our data set), time which could
already be spent loading data into the cache, once the first queries are available.
> The suggestion for improvement is to change the signature of the {{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}}
and {{org.apache.ignite.IgniteCache#loadCache}} method to use a Producer/Stream for the custom
queries, and the workers in the thread pool doing the actual loading should just get queries
from this stream until it is exhausted/get from the producer until it returns null (or some
other indicator that there are no more queries to be produced). This would shorten the total
time required to load the cache, as it would make it more similar in behavior to the part
of {{org.apache.ignite.cache.store.jdbc.CacheAbstractJdbcStore#loadCache}} handling the case
without custom queries (where as soon as the boundaries of a range are retrieved from the
initial {{loadCacheSelRangeQry}}, the load of the range is automatically submitted to the
thread pool for execution) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message