ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From nbruce9 <nbru...@bloomberg.net>
Subject High latency using affinityCall from client
Date Wed, 10 Feb 2016 15:03:51 GMT
Hi!

Sorry for the long question in advance...

I'm doing some performance measurements on Ignite to see if it would work
for a new project. So far, I love Ignite, but I would like some help
resolving an issue with latency.

Here's my setup:
3 RHEL machines running Ignite in "server" mode, with a single partitioned
cache
1 RHEL machine running Ignite in "client" mode.

>From my Ignite client I use affinityCall() to co-locate with a specific
cache key on one of the 3 servers.
I first insert 100,000 unique keys into the cache using affinityCall().
Then, I asynchronously affinityCall() to do 10 get/modify/put on each key in
parallel (with transactions), for all 100,000 keys.

For the transactional workload, I see ok throughput (around 22k
transactions/sec).
However, my average latency for the affinityCall() is unbearably high (at
67.9 ms). I am measuring this latency in my Ignite client from right before
affinityCall() to when the future's listener runs. 
In the server nodes, I see the average time taken by my call() function to
be around 1.5 ms. That means that my own code that is executing on the
server within call() is not the issue. All machines are in the same data
center, and ping is around 0.25 ms between each of the 4 machines, so I
don't believe network latency is the issue either. I also see the min
latency is great, at around 240 microseconds. There's a big spread. What
would cause such variation in latency and such a high average?

Latency: Avg=67.921748 Min=0.240000 Max=1259.326000 StdDev=83.476537

Any help or suggestions would be greatly appreciated!

Here is my ignite config and cache config:

IgniteConfiguration [gridName=null, pubPoolSize=48, sysPoolSize=48,
mgmtPoolSize=4, igfsPoolSize=24, utilityCachePoolSize=48,
utilityCacheKeepAliveTime=10000, marshCachePoolSize=48,
marshCacheKeepAliveTime=10000, p2pPoolSize=2, ggHome=null, ggWork=null,
mbeanSrv=null, nodeId=null, marsh=null, marshLocJobs=false, daemon=false,
p2pEnabled=false, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3,
clockSyncSamples=8, clockSyncFreq=120000, metricsHistSize=10000,
metricsUpdateFreq=2000, metricsExpTime=9223372036854775807,
discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0,
reconCnt=10, maxAckTimeout=600000, forceSrvMode=false,
clientReconnectDisabled=false], segPlc=STOP, segResolveAttempts=2,
waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000,
commSpi=null, evtSpi=null, colSpi=null, deploySpi=null, swapSpaceSpi=null,
indexingSpi=null, addrRslvr=null, clientMode=false,
rebalanceThreadPoolSize=1,
txCfg=org.apache.ignite.configuration.TransactionConfiguration@39a054a5,
cacheSanityCheckEnabled=false, discoStartupDelay=60000, deployMode=PRIVATE,
p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100,
timeSrvPortRange=100, failureDetectionTimeout=10000, metricsLogFreq=60000,
hadoopCfg=null,
connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@71bc1ae4,
warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000,
cacheMode=PARTITIONED, backups=0], classLdr=null, sslCtxFactory=null,
platformCfg=null, binaryCfg=null]

CacheConfiguration [name=Orders, storeConcurrentLoadAllThreshold=5,
rebalancePoolSize=2, rebalanceTimeout=10000, evictPlc=null, evictSync=false,
evictKeyBufSize=1024, evictSyncConcurrencyLvl=4, evictSyncTimeout=10000,
evictFilter=null, evictMaxOverflowRatio=10.0, eagerTtl=true,
dfltLockTimeout=0, startSize=100000, nearCfg=null, writeSync=PRIMARY_SYNC,
storeFactory=null, keepBinaryInStore=false, loadPrevVal=false, aff=null,
cacheMode=PARTITIONED, atomicityMode=TRANSACTIONAL,
atomicWriteOrderMode=CLOCK, backups=0, invalidate=false,
tmLookupClsName=null, rebalanceMode=NONE, rebalanceOrder=0,
rebalanceBatchSize=524288, rebalanceBatchesPrefetchCount=2,
offHeapMaxMem=-1, swapEnabled=false, maxConcurrentAsyncOps=1000000,
writeBehindEnabled=false, writeBehindFlushSize=10240,
writeBehindFlushFreq=5000, writeBehindFlushThreadCnt=1,
writeBehindBatchSize=512, memMode=OFFHEAP_VALUES, affMapper=null,
rebalanceDelay=0, rebalanceThrottle=0, interceptor=null,
longQryWarnTimeout=3000, readFromBackup=true, nodeFilter=null,
sqlEscapeAll=false, sqlOnheapRowCacheSize=10240, snapshotableIdx=false,
cpOnRead=true, topValidator=null]




--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/High-latency-using-affinityCall-from-client-tp2931.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message