ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ramzinator <rami.hamm...@gmail.com>
Subject Performance issue with Replicated Cache among 7 nodes
Date Mon, 15 May 2017 13:27:15 GMT
Hi all,

I have an ignite cluster of 7 nodes and a cache that is replicated among all
of them configured as follows:
<bean class="org.apache.ignite.configuration.CacheConfiguration">
          <property name="name" value="cacheName"/>
          <property name="cacheMode" value="REPLICATED"/>
          <property name="writeSynchronizationMode" value="FULL_SYNC"/>
          <property name="copyOnRead" value="false"/>
          <property name="backups" value="0"/>
</bean>

When inserting the first object (of size ~15mb) into the cache, the
insertion takes around *30 mins*.

No exception are thrown during this time, and the nodes were practically
idle. However we do witness logs such as:

2017-05-15 12:28:55.507  WARN [gridname,,,] 1 --- [gridname%]
.i.p.c.GridCachePartitionExchangeManager : Found long running cache
operations, dump IO statistics.
2017-05-15 12:28:55.509  WARN [gridname,,,] 1 --- [gridname%]
o.a.i.s.c.tcp.TcpCommunicationSpi        : Communication SPI recovery
descriptors:
    [key=ConnectionKey [nodeId=8d24da8d-38a9-455b-a295-e735bbb503e0, idx=0,
connCnt=-1], msgsSent=2, msgsAckedByRmt=2, msgsRcvd=2, lastAcked=2,
reserveCnt=5, descIdHash=420108385]
    [key=ConnectionKey [nodeId=89c5e965-6351-4878-b78c-813ae159cc29, idx=0,
connCnt=-1], msgsSent=11, msgsAckedByRmt=11, msgsRcvd=9, lastAcked=9,
reserveCnt=4, descIdHash=510995069]
    [key=ConnectionKey [nodeId=7fe63f13-d34d-4a75-8b99-233ed21f192c, idx=0,
connCnt=-1], msgsSent=7, msgsAckedByRmt=6, msgsRcvd=6, lastAcked=5,
reserveCnt=8, descIdHash=334727389]
    [key=ConnectionKey [nodeId=ac0a6e04-8ea1-427e-989f-bec85f476bb1, idx=0,
connCnt=-1], msgsSent=32, msgsAckedByRmt=32, msgsRcvd=30, lastAcked=29,
reserveCnt=4, descIdHash=1989271882]
    [key=ConnectionKey [nodeId=95ac3c7b-0445-492d-9bd5-0f22544b15f5, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=1, msgsRcvd=3, lastAcked=1,
reserveCnt=8, descIdHash=330160523]
    [key=ConnectionKey [nodeId=47f1b382-5acb-4882-b3d8-d5878857778f, idx=0,
connCnt=-1], msgsSent=9, msgsAckedByRmt=9, msgsRcvd=6, lastAcked=4,
reserveCnt=3, descIdHash=1024681641]
    [key=ConnectionKey [nodeId=98855451-c7ca-4b40-bf50-c8ae17a88a61, idx=0,
connCnt=-1], msgsSent=3, msgsAckedByRmt=2, msgsRcvd=3, lastAcked=2,
reserveCnt=8, descIdHash=167941556]
    [key=ConnectionKey [nodeId=5a079141-43ad-406c-820b-cebaf32c9729, idx=0,
connCnt=0], msgsSent=17, msgsAckedByRmt=17, msgsRcvd=14, lastAcked=13,
reserveCnt=1, descIdHash=87576905]
    [key=ConnectionKey [nodeId=573f277b-8a2c-47cc-ab0f-33d389a6294f, idx=0,
connCnt=-1], msgsSent=12, msgsAckedByRmt=12, msgsRcvd=8, lastAcked=7,
reserveCnt=4, descIdHash=2094693141]

Also the insertion completes but after a very long time.

My question is why is the insertion taking so long?
Is ignite stuck in a transaction and waiting to timeout?
I'd appreciate any help/advice on how to proceed to speed up the insertion.
Note that the issue became noticeable when we scaled up the cluster from 3
ignite nodes to 7.

Thanks,
Ramzinator



--
View this message in context: http://apache-ignite-users.70518.x6.nabble.com/Performance-issue-with-Replicated-Cache-among-7-nodes-tp12848.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

Mime
View raw message