ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Denis Magda <dma...@apache.org>
Subject Re: read-though tutorial for a big table
Date Tue, 23 Jun 2020 23:31:29 GMT
Hello Vladimir,

Sounds interesting, thanks for reaching out. Let me introduce you to @Kseniya
Romanova <kromanova@gridgain.com> who can help with the publication process.

-
Denis


On Sun, Jun 21, 2020 at 10:31 PM Vladimir Tchernyi <vtchernyi@gmail.com>
wrote:

> Hi Denis,
>
> Some progress had happened and I have some material to share with the
> community. I think it will be interesting to newbies. It is about loading
> big tables from rdbms and creating cache entries based on table info. This
> approach was tested in production and showed good timing being paired with
> MSSQL, tables from tens to hundreds million rows.
>
> The loading jar process:
> * starts Ignite client node;
> * creates user POJO according to business logic;
> * converts POJOs to BinaryObjects;
> * uses affinity function and creates separate key-value HashMap for every
> cache partition;
> * uses ComputeTaskAdaper/ComputeJobAdaper to place hashMaps on
> corresponding data node.
>
> I would like to publish some tutorial, say on GridGain website in english
> and russian version on habr.com.
>
> WDYT?
>
> чт, 12 мар. 2020 г. в 08:25, <vtchernyi@gmail.com>:
>
>> Hello Denis,
>>
>> That is possible, my writing activities should be continued. The only
>> question is to get my local project to production, there is no sense in
>> writing another model example. So I hope there will be a progress in the
>> nearest future
>>
>> Vladimir
>>
>> 2:25, 12 марта 2020 г., Denis Magda <dmagda@apache.org>:
>>
>> Hello Vladimir,
>>
>> Just to clarify, are you suggesting to create a tutorial for data loading
>> scenarios when data resides in an external database?
>>
>> -
>> Denis
>>
>>
>> On Tue, Mar 10, 2020 at 11:41 PM <vtchernyi@gmail.com> wrote:
>>
>> Andrei, Evgenii, thanks for answer.
>>
>> Aa far as I see, there is no ready to use tutorial. I managed to do
>> multi-threaded cache load procedure, out-of-the-box loadCache method is
>> extremely slow.
>>
>> I spent about a month studying write-through topics, and finally got the
>> same as "capacity planning" says: 0.8Gb mssql table on disk expands to
>> 2.3Gb, size in ram is 2.875 times bigger.
>>
>> Is it beneficial to use BinaryObject instead of user pojo? If yes, how to
>> create BinaryObject without pojo definition and deserialize it back to pojo?
>> It would be great to have kind of advanced github example like this
>>
>> https://github.com/dmagda/MicroServicesExample
>>
>> It helped a lot in understanding. Current documentation links do not help
>> to build a real solution, they are mostly like a reference, with no option
>> to compile and debug
>>
>> Vladimir
>>
>> 2:51, 11 марта 2020 г., Evgenii Zhuravlev <e.zhuravlev.wk@gmail.com>:
>>
>> When you're saying that the result was poor, do you mean that data
>> preloading took too much time, or it's just about get operations?
>>
>> Evgenii
>>
>> вт, 10 мар. 2020 г. в 03:29, aealexsandrov <aealexsandrov@gmail.com>:
>>
>> Hi,
>>
>> You can read the documentation articles:
>>
>> https://apacheignite.readme.io/docs/3rd-party-store
>>
>> In case if you are going to load the cache from 3-rd party store (RDBMS)
>> then the default implementation of CacheJdbcPojoStore can take a lot of
>> time
>> for loading the data because it used JDBC connection inside (not pull of
>> these connections).
>>
>> Probably you should implement your own version of CacheStore that will
>> read
>> data from RDBMS in several threads, e.g using the JDBC connection pull
>> there. Sources are open for you, so you can copy the existed
>> implementation
>> and modify it:
>>
>>
>> https://github.com/apache/ignite/blob/master/modules/core/src/main/java/org/apache/ignite/cache/store/jdbc/CacheJdbcPojoStore.java
>>
>> Otherwise, you can do the initial data loading using some streaming tools:
>>
>> 1)Spark integration with Ignite -
>> https://apacheignite-fs.readme.io/docs/ignite-data-frame
>> 2)Kafka integration with Ignite -
>> https://apacheignite-mix.readme.io/docs/kafka-streamer
>>
>> BR,
>> Andrei
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>>
>>
>> --
>> Отправлено из мобильного приложения Яндекс.Почты
>>
>>
>>
>> --
>> Отправлено из мобильного приложения Яндекс.Почты
>
>

Mime
View raw message