ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ray <ray...@cisco.com>
Subject Re: Inserting data into Ignite got stuck when memory is full with persistent store enabled.
Date Tue, 17 Oct 2017 14:09:29 GMT
I'm using ignite 2.1.

The phenomenon I observed is that for the first 130M entries the speed is
OK, but after about 130M entries it slowed down tremendously and finally it
will stuck.
The problem is that when I ingest small amount of data like 20M, it works OK
and the performance is acceptable.
But when the entries number grow, the issue will happen for sure.
Please try ingesting more data and see if you can reproduce this issue.

My servers have HDD, and I tested the random write speed using this command
" fio -filename=/dev/sda -direct=1 -iodepth 1 -thread -rw=randwrite
-ioengine=psync -bs=16k -size=200G -numjobs=30 -runtime=1000
-group_reporting -name=mytest
"
The result is
mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync,
iodepth=1
...
mytest: (g=0): rw=randwrite, bs=16K-16K/16K-16K/16K-16K, ioengine=psync,
iodepth=1
fio-2.0.13
Starting 30 threads
Jobs: 11 (f=11): [__wwww_ww__w_w___w____w______w] [5.0% done] [0K/3072K/0K
/s] [0 /192 /0  iops] [eta 05h:16m:40s]
mytest: (groupid=0, jobs=30): err= 0: pid=17321: Tue Oct 17 13:54:12 2017
  write: io=5835.2MB, bw=5969.5KB/s, iops=373 , runt=1000960msec
    clat (usec): min=149 , max=2013.4K, avg=80349.31, stdev=53474.06
     lat (usec): min=150 , max=2013.4K, avg=80351.00, stdev=53474.13
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[   10], 10.00th=[   14], 20.00th=[   48],
     | 30.00th=[   67], 40.00th=[   78], 50.00th=[   86], 60.00th=[   92],
     | 70.00th=[   99], 80.00th=[  109], 90.00th=[  123], 95.00th=[  139],
     | 99.00th=[  180], 99.50th=[  215], 99.90th=[  424], 99.95th=[  611],
     | 99.99th=[ 2008]
    bw (KB/s)  : min=    7, max= 8440, per=3.36%, avg=200.43, stdev=72.14
    lat (usec) : 250=0.01%, 500=0.04%, 750=0.05%, 1000=0.01%
    lat (msec) : 2=0.09%, 4=1.71%, 10=3.23%, 20=9.55%, 50=5.60%
    lat (msec) : 100=50.94%, 250=28.38%, 500=0.33%, 750=0.02%, 1000=0.01%
    lat (msec) : 2000=0.01%, >=2000=0.02%
  cpu          : usr=0.00%, sys=0.00%, ctx=331635, majf=0,
minf=18446744073700073665
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%,
>=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%,
>=64=0.0%
     issued    : total=r=0/w=373449/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=5835.2MB, aggrb=5969KB/s, minb=5969KB/s, maxb=5969KB/s,
mint=1000960msec, maxt=1000960msec

Disk stats (read/write):
  sda: ios=166/373605, merge=0/108, ticks=203/30925452, in_queue=30935375,
util=100.00%



The estimated field length for key+value is 300 bytes each entry.
So the total amount will be write to disc will be
(300+300)*1.3*550*1000*1000 bytes = 426GB
Because I have four ignite servers, so the total time will be 426 / 4 /
(5969 / 1000 / 1000) / 60 / 60 = 5 hours.

So apparently the disk writing speed is not an issue, it took ignite more
13.5 hours to ingest 250M entries.

By the way, I launched six ignite clients on the same node ignite server
runs.








--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Mime
View raw message