ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nathaniel Bruce (BLOOMBERG/ 731 LEX)" <nbru...@bloomberg.net>
Subject Re:High latency using affinityCall from client
Date Wed, 10 Feb 2016 19:58:42 GMT
Hi again,

So my issue seems to be due to saturation at the client side. If I do too many async operations,
they appear to be queueing up, so the latency I was seeing included the time they were waiting
in line to go out to the server.

Is it possible to increase the number of parallel async operations in my client? Is there
config for setting a threadpool size of the outgoing async operations? I tried setting MaxConcurrentAsyncOperations
on the cache config but that had no effect. 


From: user@ignite.apache.org At: Feb 10 2016 10:15:37
To: user@ignite.apache.org
Subject: Re:High latency using affinityCall from client

Hi!

Sorry for the long question in advance...

I'm doing some performance measurements on Ignite to see if it would work
for a new project. So far, I love Ignite, but I would like some help
resolving an issue with latency.

Here's my setup:
3 RHEL machines running Ignite in "server" mode, with a single partitioned
cache
1 RHEL machine running Ignite in "client" mode.

From my Ignite client I use affinityCall() to co-locate with a specific
cache key on one of the 3 servers.
I first insert 100,000 unique keys into the cache using affinityCall().
Then, I asynchronously affinityCall() to do 10 get/modify/put on each key in
parallel (with transactions), for all 100,000 keys.

For the transactional workload, I see ok throughput (around 22k
transactions/sec).
However, my average latency for the affinityCall() is unbearably high (at
67.9 ms). I am measuring this latency in my Ignite client from right before
affinityCall() to when the future's listener runs. 
In the server nodes, I see the average time taken by my call() function to
be around 1.5 ms. That means that my own code that is executing on the
server within call() is not the issue. All machines are in the same data
center, and ping is around 0.25 ms between each of the 4 machines, so I
don't believe network latency is the issue either. I also see the min
latency is great, at around 240 microseconds. There's a big spread. What
would cause such variation in latency and such a high average?

Latency: Avg=67.921748 Min=0.240000 Max=1259.326000 StdDev=83.476537

Any help or suggestions would be greatly appreciated!

Here is my ignite config and cache config:

IgniteConfiguration [gridName=null, pubPoolSize=48, sysPoolSize=48,
mgmtPoolSize=4, igfsPoolSize=24, utilityCachePoolSize=48,
utilityCacheKeepAliveTime=10000, marshCachePoolSize=48,
marshCacheKeepAliveTime=10000, p2pPoolSize=2, ggHome=null, ggWork=null,
mbeanSrv=null, nodeId=null, marsh=null, marshLocJobs=false, daemon=false,
p2pEnabled=false, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3,
clockSyncSamples=8, clockSyncFreq=120000, metricsHistSize=10000,
metricsUpdateFreq=2000, metricsExpTime=9223372036854775807,
discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0,
reconCnt=10, maxAckTimeout=600000, forceSrvMode=false,
clientReconnectDisabled=false], segPlc=STOP, segResolveAttempts=2,
waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000,
commSpi=null, evtSpi=null, colSpi=null, deploySpi=null, swapSpaceSpi=null,
indexingSpi=null, addrRslvr=null, clientMode=false,
rebalanceThreadPoolSize=1,
txCfg=org.apache.ignite.configuration.TransactionConfiguration@39a054a5,
cacheSanityCheckEnabled=false, discoStartupDelay=60000, deployMode=PRIVATE,
p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100,
timeSrvPortRange=100, failureDetectionTimeout=10000, metricsLogFreq=60000,
hadoopCfg=null,
connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@71bc1ae4,
warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000,
cacheMode=PARTITIONED, backups=0], classLdr=null, sslCtxFactory=null,
platformCfg=null, binaryCfg=null]

CacheConfiguration [name=Orders, storeConcurrentLoadAllThreshold=5,
rebalancePoolSize=2, rebalanceTimeout=10000, evictPlc=null, evictSync=false,
evictKeyBufSize=1024, evictSyncConcurrencyLvl=4, evictSyncTimeout=10000,
evictFilter=null, evictMaxOverflowRatio=10.0, eagerTtl=true,
dfltLockTimeout=0, startSize=100000, nearCfg=null, writeSync=PRIMARY_SYNC,
storeFactory=null, keepBinaryInStore=false, loadPrevVal=false, aff=null,
cacheMode=PARTITIONED, atomicityMode=TRANSACTIONAL,
atomicWriteOrderMode=CLOCK, backups=0, invalidate=false,
tmLookupClsName=null, rebalanceMode=NONE, rebalanceOrder=0,
rebalanceBatchSize=524288, rebalanceBatchesPrefetchCount=2,
offHeapMaxMem=-1, swapEnabled=false, maxConcurrentAsyncOps=1000000,
writeBehindEnabled=false, writeBehindFlushSize=10240,
writeBehindFlushFreq=5000, writeBehindFlushThreadCnt=1,
writeBehindBatchSize=512, memMode=OFFHEAP_VALUES, affMapper=null,
rebalanceDelay=0, rebalanceThrottle=0, interceptor=null,
longQryWarnTimeout=3000, readFromBackup=true, nodeFilter=null,
sqlEscapeAll=false, sqlOnheapRowCacheSize=10240, snapshotableIdx=false,
cpOnRead=true, topValidator=null]


--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/High-latency-using-affinityCall-from-client-tp2931.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.


Mime
View raw message