ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ozerov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-3401) "Remote node ID is not as expected" errors at driver during load test
Date Fri, 09 Dec 2016 12:48:01 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-3401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vladimir Ozerov updated IGNITE-3401:
------------------------------------
    Fix Version/s:     (was: 1.9)
                   2.0

> "Remote node ID is not as expected" errors at driver during load test
> ---------------------------------------------------------------------
>
>                 Key: IGNITE-3401
>                 URL: https://issues.apache.org/jira/browse/IGNITE-3401
>             Project: Ignite
>          Issue Type: Bug
>          Components: general
>            Reporter: Ksenia Rybakova
>             Fix For: 2.0
>
>
> A lot of errors at some drivers after several minutes after grid is up during load test:
> {noformat}
> [14:33:42] (err) Failed to execute compound future reducer: Compound future listener
[]class org.apache.ignite.IgniteCheckedException: Failed to send message (node may have left
the grid or TCP connection cannot be established due to fir
>         at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1129)
>         at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1193)
>         at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.submit(DataStreamerImpl.java:1378)
>         at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$Buffer.update(DataStreamerImpl.java:1173)
>         at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.load0(DataStreamerImpl.java:756)
>         at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addDataInternal(DataStreamerImpl.java:568)
>         at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl.addData(DataStreamerImpl.java:604)
>         at org.apache.ignite.yardstick.cache.load.IgniteCacheRandomOperationBenchmark$1.run(IgniteCacheRandomOperationBenchmark.java:383)
> Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to send message to
remote node: TcpDiscoveryNode [id=11da20f5-87c5-4321-8163-ffef85f2d54c, addrs=[10.20.0.215,
127.0.0.1], sockAddrs=[/127.0.0.1:47503, fosters-215/10.20.0
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1986)
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:1926)
>         at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1124)
>         ... 7 more
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to connect to node
(is node still alive?). Make sure that each GridComputeTask and GridCacheTransaction has a
timeout set in order to prevent parties from waiting forever
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2489)
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2130)
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2024)
>         at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:1960)
>         ... 9 more
>         Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect
to address: fosters-215/10.20.0.215:47103
>                 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2494)
>                 ... 12 more
>         Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read remote
node recovery handshake (connection closed).
>                 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2699)
>                 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2361)
>                 ... 12 more
>         Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to connect
to address: /127.0.0.1:47103
>                 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2494)
>                 ... 12 more
>         Caused by: class org.apache.ignite.IgniteCheckedException: Remote node ID is
not as expected [expected=11da20f5-87c5-4321-8163-ffef85f2d54c, rcvd=03bc7d54-4fec-481d-af24-6f5408b32fe2]
>                 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.safeHandshake(TcpCommunicationSpi.java:2604)
>                 at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2361)
>                 ... 12 more
> {noformat}
> Tested version: 1.5.27
> Load config:
> - 7 hosts with 4 servers and 3 drivers each (28 servers and 21 drivers in total)
> - benchmark: IgniteCacheRandomOperationBenchmark
> - preload: 2M
> - key range: 4M
> - warmup: 30s
> - duration: 10h 
> - operations: PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL PUT_IF_ABSENT
REPLACE
> -backup count: 1
> - GC: G1
> - heap size: 8Gb for servers, 4Gb for drivers
> - caches: 
> {noformat}
>     <bean name="atomic" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="atomic"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="ATOMIC"/>
>         <property name="swapEnabled" value="false"/>
>     </bean>
>     <bean name="atomic-evict" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="atomic-evict"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="ATOMIC"/>
>         <property name="swapEnabled" value="false"/>
>         <property name="offHeapMaxMemory" value="#{1024L*1024L*1024L*6L}"/>
>         <property name="evictionPolicy">
>             <bean class="org.apache.ignite.cache.eviction.fifo.FifoEvictionPolicy">
>                 <property name="maxSize" value="750000"/>
>             </bean>
>         </property>
>     </bean>
>     <bean name="atomic-offheap" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="atomic-offheap"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="ATOMIC"/>
>         <property name="swapEnabled" value="false"/>
>         <property name="memoryMode" value="OFFHEAP_TIERED"/>
>     </bean>
>     <bean name="atomic-offheap-values" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="atomic-offheap-values"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="ATOMIC"/>
>         <property name="swapEnabled" value="false"/>
>         <property name="memoryMode" value="OFFHEAP_VALUES"/>
>     </bean>
>     <bean name="tx" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="tx"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="TRANSACTIONAL"/>
>         <property name="swapEnabled" value="false"/>
>     </bean>
>     <bean name="tx-evict" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="tx-evict"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="TRANSACTIONAL"/>
>         <property name="swapEnabled" value="false"/>
>         <property name="offHeapMaxMemory" value="#{1024L*1024L*1024L*6L}"/>
>         <property name="evictionPolicy">
>             <bean class="org.apache.ignite.cache.eviction.lru.LruEvictionPolicy">
>                 <property name="maxSize" value="750000"/>
>             </bean>
>         </property>
>     </bean>
>     <bean name="compute" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="compute"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="TRANSACTIONAL"/>
>         <property name="swapEnabled" value="false"/>
>         <property name="offHeapMaxMemory" value="#{1024L*1024L*1024L*6L}"/>
>         <property name="evictionPolicy">
>             <bean class="org.apache.ignite.cache.eviction.fifo.FifoEvictionPolicy">
>                 <property name="maxSize" value="750000"/>
>             </bean>
>         </property>
>     </bean>
>     <bean name="tx-offheap" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="tx-offheap"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="TRANSACTIONAL"/>
>         <property name="swapEnabled" value="false"/>
>         <property name="memoryMode" value="OFFHEAP_TIERED"/>
>     </bean>
>     <bean name="tx-offheap-values" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="tx-offheap-values"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="TRANSACTIONAL"/>
>         <property name="swapEnabled" value="false"/>
>         <property name="memoryMode" value="OFFHEAP_VALUES"/>
>     </bean>
>     <bean name="atomic-index" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="atomic-index"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="ATOMIC"/>
>         <property name="swapEnabled" value="false"/>
>         <property name="indexedTypes">
>             <list>
>                 <value>java.lang.Integer</value>
>                <value>org.apache.ignite.yardstick.cache.model.Person1</value>
>             </list>
>         </property>
>     </bean>
>     <bean name="query" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="query"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="ATOMIC"/>
>         <property name="swapEnabled" value="false"/>
>         <!-- Configure query entities -->
>         <property name="queryEntities">
>             <list>
>                 <bean class="org.apache.ignite.cache.QueryEntity">
>                     <property name="keyType" value="java.lang.Integer"/>
>                     <property name="valueType" value="org.apache.ignite.yardstick.cache.model.Person"/>
>                     <property name="fields">
>                         <map>
>                             <entry key="id" value="java.lang.Integer"/>
>                             <entry key="orgId" value="java.lang.Integer"/>
>                             <entry key="firstName" value="java.lang.String"/>
>                             <entry key="lastName" value="java.lang.String"/>
>                             <entry key="salary" value="java.lang.Double"/>
>                         </map>
>                     </property>
>                     <property name="indexes">
>                         <list>
>                             <bean class="org.apache.ignite.cache.QueryIndex">
>                                 <constructor-arg value="id"/>
>                             </bean>
>                             <!-- Group index. -->
>                             <bean class="org.apache.ignite.cache.QueryIndex">
>                                 <constructor-arg>
>                                     <list>
>                                         <value>id</value>
>                                         <value>salary</value>
>                                     </list>
>                                 </constructor-arg>
>                                 <constructor-arg value="SORTED"/>
>                             </bean>
>                         </list>
>                     </property>
>                 </bean>
>             </list>
>         </property>
>     </bean>
>     <bean name="orgCache" class="org.apache.ignite.configuration.CacheConfiguration">
>         <property name="name" value="orgCache"/>
>         <property name="cacheMode" value="PARTITIONED"/>
>         <property name="atomicityMode" value="ATOMIC"/>
>         <property name="swapEnabled" value="false"/>
>         <!-- Configure query entities -->
>         <property name="queryEntities">
>             <list>
>                 <bean class="org.apache.ignite.cache.QueryEntity">
>                     <property name="keyType" value="java.lang.Integer"/>
>                     <property name="valueType" value="org.apache.ignite.yardstick.cache.model.Organization"/>
>                     <property name="fields">
>                         <map>
>                             <entry key="id" value="java.lang.Integer"/>
>                             <entry key="name" value="java.lang.String"/>
>                         </map>
>                     </property>
>                     <property name="indexes">
>                         <list>
>                             <bean class="org.apache.ignite.cache.QueryIndex">
>                                 <constructor-arg value="id"/>
>                             </bean>
>                             <bean class="org.apache.ignite.cache.QueryIndex">
>                                 <constructor-arg value="name"/>
>                             </bean>
>                         </list>
>                     </property>
>                 </bean>
>             </list>
>         </property>
>     </bean>
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message