ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Saifullah Zahid <saif.m...@gmail.com>
Subject Re: Ignite.NET very slow Cache Loading
Date Thu, 16 Mar 2017 12:23:56 GMT
Hi Pavel,

Thanks for  your reply.
There are about 40 fileds TransactionsDetail and size is about 400 bytes.
In Taskbar I can see the process is taking about 23GB
I used 20GB heap-size.

Also I have noticed that c: drive have only 3GB empty space.

Thanks,
Saif

On Thu, Mar 16, 2017 at 11:16 AM, Pavel Tupitsyn <ptupitsyn@apache.org>
wrote:

> I thought that you use DataStreamer to load data into cache for some
> reason.
> As I now see in gitter, you actually use ICacheStore.LoadCache.
>
> So my next thought is that slowdown can be caused by GC because of low
> memory.
> How much memory one row takes? Can you show TransactionsDetail code, does
> it have a lot of fields?
> Can you attach to your process with jvisualvm.exe (part of Java JDK) and
> see how much heap memory is actually used?
> Another option is to look at Private Working Set in Task Manager.
>
> Please also attach Ignite log.
>
> On Wed, Mar 15, 2017 at 8:13 PM, Saifullah Zahid <saif.muet@gmail.com>
> wrote:
>
>> I tried that but it is throwing exception
>>
>> class org.apache.ignite.IgniteCheckedException: Cannot enable
>> write-behind (writer or store is not provided) for cache:
>>
>> If I comment following settings
>>
>> ReadThrough = true,
>> WriteThrough = true,
>> WriteBehindEnabled = true,
>>
>> cache gets created but without loading data from database.
>>
>> thanks,
>>
>>
>>
>> On Wed, Mar 15, 2017 at 10:00 PM, Pavel Tupitsyn <ptupitsyn@apache.org>
>> wrote:
>>
>>> Why use Spring?
>>> I propose to do just one thing - comment out the CacheStoreFactory line.
>>>
>>> On Wed, Mar 15, 2017 at 7:30 PM, Saifullah Zahid <saif.muet@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> If I disable CacheStoreFactor then I have to provide the
>>>> CacheConfiguration in spring xml
>>>> But I don't know how to provide Database and table mapping in spring
>>>> xml, I tried following configuration but it is failing.
>>>> I am not sure if it is right way to do?
>>>>
>>>> <?xml version="1.0" encoding="UTF-8"?>
>>>> <beans xmlns="http://www.springframework.org/schema/beans"
>>>>        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>>>        xmlns:util="http://www.springframework.org/schema/util"
>>>>        xsi:schemaLocation="http://www.springframework.org/schema/beans
>>>>                            http://www.springframework.or
>>>> g/schema/beans/spring-beans.xsd
>>>>                            http://www.springframework.org/schema/util
>>>>                            http://www.springframework.or
>>>> g/schema/util/spring-util.xsd">
>>>>
>>>>     <bean id="ignite.cfg" class="org.apache.ignite.confi
>>>> guration.IgniteConfiguration">
>>>> <bean id="myDataSource" class="org.springframework.jdb
>>>> c.datasource.DriverManagerDataSource">
>>>> <property name="driverClassName" value="com.microsoft.sqlserver.jdbc.SQLServerDriver"
>>>> />
>>>> <property name="url" value="jdbc:sqlserver://localh
>>>> ost:1433;databaseName=TransactionsDB" />
>>>> <property name="username" value="user1" />
>>>> <property name="password" value="password123" />
>>>> </bean>
>>>>  <!-- Enabling ODBC. -->
>>>>  <property name="odbcConfiguration">
>>>> <bean class="org.apache.ignite.configuration.OdbcConfiguration"/>
>>>>  </property>
>>>> </bean>
>>>> <property name="cacheConfiguration">
>>>> <list>
>>>> <bean class="org.apache.ignite.configuration.CacheConfiguration">
>>>> <property name="dataSourceBean" value="myDataSource" />
>>>> <property name="databaseSchema" value="dbo" />
>>>>                     <property name="name" value="transactionsdetail"/>
>>>>                     <property name="databaseTable"
>>>> value="transactionsdetail" />
>>>>             </bean>
>>>> </list>
>>>> </property>
>>>> </beans>
>>>>
>>>> Getting following error
>>>>
>>>>  Failed to instantiate configuration from Spring XML
>>>>
>>>> Thanks,
>>>> Saif
>>>>
>>>> On Wed, Mar 15, 2017 at 8:29 PM, Pavel Tupitsyn <ptupitsyn@apache.org>
>>>> wrote:
>>>>
>>>>> Have you tried commenting out cache store as I descrived above?
>>>>>
>>>>> On Wed, Mar 15, 2017 at 6:10 PM, Saifullah Zahid <saif.muet@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> following is my load cache method
>>>>>> if I comments out following line of code, it does not slows down
>>>>>> act(GetKeyValue(item), item);
>>>>>>
>>>>>> public void LoadCache(Action<object, object> act, params object[]
>>>>>> args)
>>>>>> {
>>>>>> using (DbConnection _connection = Create())
>>>>>> {
>>>>>> using (DbCommand command = _connection.CreateCommand())
>>>>>> {
>>>>>> MakeLoadCacheCommand(command, args);
>>>>>> using (var reader = command.ExecuteReader())
>>>>>> {
>>>>>> while (reader.Read())
>>>>>> {
>>>>>> var item = new T();
>>>>>> Map(reader, item); //populate object
>>>>>> * act(GetKeyValue(item), item); //insert into cache*
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> Looks like issue is when inserting data into cache.
>>>>>>
>>>>>> Thanks,
>>>>>> Saif
>>>>>>
>>>>>>
>>>>>> On Wed, Mar 15, 2017 at 7:02 PM, Pavel Tupitsyn <ptupitsyn@apache.org
>>>>>> > wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> You have cache store configured, which probably causes the slowdown.
>>>>>>> Please try to disable cache store (remove CacheStoreFactory from
>>>>>>> config) and see if it makes any difference.
>>>>>>>
>>>>>>> On Wed, Mar 15, 2017 at 4:02 PM, Saifullah Zahid <
>>>>>>> saif.muet@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I am facing an issue on cache loading, In the start cache
loads
>>>>>>>> quickly but after some time it becomes very slow almost 1
row per second.
>>>>>>>> There are about 4 million rows in a table.
>>>>>>>> OS = Windows Server 2012
>>>>>>>> RAM = 64 GB
>>>>>>>> Node Heap config is 16 GB.
>>>>>>>> Following is Cache configuration
>>>>>>>>
>>>>>>>> TransactionsDetailStore = m_cache.GetOrCreateCache<int,
>>>>>>>> TransactionsDetail>(new CacheConfiguration("transactionssdetail",
>>>>>>>> typeof(TransactionsDetail))
>>>>>>>> {
>>>>>>>> CacheStoreFactory = new TransactionsDetailStoreFactory("ApplicationDB",
>>>>>>>> true),
>>>>>>>> ReadThrough = true,
>>>>>>>> WriteThrough = true,
>>>>>>>> WriteBehindEnabled = true,
>>>>>>>> KeepBinaryInStore = false,
>>>>>>>> WriteBehindFlushThreadCount = 4,
>>>>>>>> WriteBehindFlushFrequency = new TimeSpan(0, 0, 2),
>>>>>>>> MemoryMode = CacheMemoryMode.OffheapTiered,
>>>>>>>> OffHeapMaxMemory = 0,
>>>>>>>> EvictionPolicy = new LruEvictionPolicy { MaxSize = 1000000
},
>>>>>>>> WriteSynchronizationMode = CacheWriteSynchronizationMode.FullSync
>>>>>>>> });
>>>>>>>>
>>>>>>>> Am I missing some configuration?
>>>>>>>> Kindly see if anyone have idea?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Saif
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message