ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dmitry Pavlov <dpavlov....@gmail.com>
Subject Re: Inserting data into Ignite got stuck when memory is full with persistent store enabled.
Date Tue, 17 Oct 2017 15:37:14 GMT
sorry for misprint. I meant thread dumps, of course

вт, 17 окт. 2017 г. в 18:16, Dmitry Pavlov <dpavlov.spb@gmail.com>:

> Hi Ray,
>
> Thank you for your reply. In addition to checkpoint marker, setting page
> size to 4K (already default in newer versions of Ignite) and WAL history
> size to value “1” may help to reduce overhead and space used and make
> loading a little faster.
>
> I apologize if you already mentioned this, but are there any trade dumps
> for the client and servers at the time of the supposed hangup?
>
> My estimate based on first test for index size for this case is around
> 80Gb (20Gb per node). Can be data of such volume fit fully in memory? How
> much cluster RAM available?
>
> Sincerely,
> Dmitriy Pavlov
>
>
> вт, 17 окт. 2017 г. в 17:09, Ray <rayliu@cisco.com>:
>
>> I'm using ignite 2.1.
>>
>> The phenomenon I observed is that for the first 130M entries the speed is
>> OK, but after about 130M entries it slowed down tremendously and finally
>> it
>> will stuck.
>> The problem is that when I ingest small amount of data like 20M, it works
>> OK
>> and the performance is acceptable.
>> But when the entries number grow, the issue will happen for sure.
>> Please try ingesting more data and see if you can reproduce this issue.
>>
>> My servers have HDD, and I tested the random write speed using this
>> command
>> " fio -filename=/dev/sda -direct=1 -iodepth 1 -thread -rw=randwrite
>> -ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000
>> -group_reporting -name=mytest
>> "
>> The result is
>> mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync,
>> iodepth=1
>> ...
>> mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync,
>> iodepth=1
>> fio-2.0.13
>> Starting 30 threads
>> Jobs: 11 (f=11): [__wwww_ww__w_w___w____w______w] [5.0% done] [0K/3072K/0K
>> /s] [0 /192 /0  iops] [eta 05h:16m:40s]
>> mytest: (groupid=0, jobs=30): err= 0: pid=17321: Tue Oct 17 13:54:12 2017
>>   write: io=5835.2MB, bw=5969.5KB/s, iops=373 , runt=1000960msec
>>     clat (usec): min=149 , max=2013.4K, avg=80349.31, stdev=53474.06
>>      lat (usec): min=150 , max=2013.4K, avg=80351.00, stdev=53474.13
>>     clat percentiles (msec):
>>      |  1.00th=[    4],  5.00th=[   10], 10.00th=[   14], 20.00th=[   48],
>>      | 30.00th=[   67], 40.00th=[   78], 50.00th=[   86], 60.00th=[   92],
>>      | 70.00th=[   99], 80.00th=[  109], 90.00th=[  123], 95.00th=[  139],
>>      | 99.00th=[  180], 99.50th=[  215], 99.90th=[  424], 99.95th=[  611],
>>      | 99.99th=[ 2008]
>>     bw (KB/s)  : min=    7, max= 8440, per=3.36%, avg=200.43, stdev=72.14
>>     lat (usec) : 250=0.01%, 500=0.04%, 750=0.05%, 1000=0.01%
>>     lat (msec) : 2=0.09%, 4=1.71%, 10=3.23%, 20=9.55%, 50=5.60%
>>     lat (msec) : 100=50.94%, 250=28.38%, 500=0.33%, 750=0.02%, 1000=0.01%
>>     lat (msec) : 2000=0.01%, >=2000=0.02%
>>   cpu          : usr=0.00%, sys=0.00%, ctx=331635, majf=0,
>> minf=18446744073700073665
>>   IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>> >=64=0.0%
>>      submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>>      complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>> >=64=0.0%
>>      issued    : total=r=0/w=373449/d=0, short=r=0/w=0/d=0
>>
>> Run status group 0 (all jobs):
>>   WRITE: io=5835.2MB, aggrb=5969KB/s, minb=5969KB/s, maxb=5969KB/s,
>> mint=1000960msec, maxt=1000960msec
>>
>> Disk stats (read/write):
>>   sda: ios=166/373605, merge=0/108, ticks=203/30925452, in_queue=30935375,
>> util=100.00%
>>
>>
>>
>> The estimated field length for key+value is 300 bytes each entry.
>> So the total amount will be write to disc will be
>> (300+300)*1.3*550*1000*1000 bytes = 426GB
>> Because I have four ignite servers, so the total time will be 426 / 4 /
>> (5969 / 1000 / 1000) / 60 / 60 = 5 hours.
>>
>> So apparently the disk writing speed is not an issue, it took ignite more
>> 13.5 hours to ingest 250M entries.
>>
>> By the way, I launched six ignite clients on the same node ignite server
>> runs.
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>

Mime
View raw message