Hi,

Please provide logs not only from the server node, bu from the client node too. You mentioned that only one client has this problems, so, please provide full log from this node.

Also, you said that you set not default timeouts for clients, while there are still default values for server node - I wouldn't recommend to do this, timeouts should be the same for all nodes in cluster.

Evgenii

ср, 15 апр. 2020 г. в 03:04, Rajan Ahlawat <rajan.ahlawat@gmail.com>:
Shared file with email-id:
e.zhuravlev.wk@gmail.com

We have single instance of ignite, File contains all log of date Mar
30, 2019. Line 6429 is the first incident of occurrence.

On Tue, Apr 14, 2020 at 8:27 PM Evgenii Zhuravlev
<e.zhuravlev.wk@gmail.com> wrote:
>
> Can you provide full log files from all nodes? it's impossible to find the root cause from this.
>
> Evgenii
>
> вт, 14 апр. 2020 г. в 07:49, Rajan Ahlawat <rajan.ahlawat@gmail.com>:
>>
>> server starts with following configuration:
>>
>> ignite_application-1-2020-03-17.log:14:[2020-03-17T08:23:33,664][INFO
>> ][main][IgniteKernal%igniteStart] IgniteConfiguration
>> [igniteInstanceName=igniteStart, pubPoolSize=32, svcPoolSize=32,
>> callbackPoolSize=32, stripedPoolSize=32, sysPoolSize=30,
>> mgmtPoolSize=4, igfsPoolSize=32, dataStreamerPoolSize=32,
>> utilityCachePoolSize=32, utilityCacheKeepAliveTime=60000,
>> p2pPoolSize=2, qryPoolSize=32,
>> igniteHome=/home/patrochandan01/ignite/apache-ignite-fabric-2.6.0-bin,
>> igniteWorkDir=/home/patrochandan01/ignite/apache-ignite-fabric-2.6.0-bin/work,
>> mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94fa3e,
>> nodeId=53396cb7-1b66-43da-bf10-ebb5f7cc9693,
>> marsh=org.apache.ignite.internal.binary.BinaryMarshaller@42b3b079,
>> marshLocJobs=false, daemon=false, p2pEnabled=false, netTimeout=5000,
>> sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000,
>> metricsUpdateFreq=2000, metricsExpTime=9223372036854775807,
>> discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0,
>> marsh=null, reconCnt=100, reconDelay=10000, maxAckTimeout=600000,
>> forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null],
>> segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true,
>> allResolversPassReq=true, segChkFreq=10000,
>> commSpi=TcpCommunicationSpi [connectGate=null, connPlc=null,
>> enableForcibleNodeKill=false, enableTroubleshootingLog=false,
>> srvLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2@6692b6c6,
>> locAddr=null, locHost=null, locPort=47100, locPortRange=100,
>> shmemPort=-1, directBuf=true, directSndBuf=false,
>> idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000,
>> reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=1024,
>> slowClientQueueLimit=1000, nioSrvr=null, shmemSrv=null,
>> usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true,
>> filterReachableAddresses=false, ackSndThreshold=32,
>> unackedMsgsBufSize=0, sockWriteTimeout=2000, lsnr=null,
>> boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=16,
>> selectorSpins=0, addrRslvr=null,
>> ctxInitLatch=java.util.concurrent.CountDownLatch@1cd629b3[Count = 1],
>> stopping=false,
>> metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@589da3f3],
>> evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@39d76cb5,
>> colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [lsnr=null],
>> indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@1cb346ea,
>> addrRslvr=null, clientMode=false, rebalanceThreadPoolSize=1,
>> txCfg=org.apache.ignite.configuration.TransactionConfiguration@4c012563,
>> cacheSanityCheckEnabled=true, discoStartupDelay=60000,
>> deployMode=SHARED, p2pMissedCacheSize=100, locHost=null,
>> timeSrvPortBase=31100, timeSrvPortRange=100,
>> failureDetectionTimeout=10000, clientFailureDetectionTimeout=30000,
>> metricsLogFreq=60000, hadoopCfg=null,
>> connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@14a50707,
>> odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration
>> [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null,
>> grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null,
>> binaryCfg=null, memCfg=null, pstCfg=null,
>> dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040,
>> sysCacheMaxSize=104857600, pageSize=0, concLvl=25,
>> dfltDataRegConf=DataRegionConfiguration [name=Default_Region,
>> maxSize=20971520, initSize=15728640, swapPath=null,
>> pageEvictionMode=RANDOM_2_LRU, evictionThreshold=0.9,
>> emptyPagesPoolSize=100, metricsEnabled=false,
>> metricsSubIntervalCount=5, metricsRateTimeInterval=60000,
>> persistenceEnabled=false, checkpointPageBufSize=0], storagePath=null,
>> checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4,
>> checkpointWriteOrder=SEQUENTIAL, walHistSize=20, walSegments=10,
>> walSegmentSize=67108864, walPath=db/wal,
>> walArchivePath=db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY,
>> walTlbSize=131072, walBuffSize=0, walFlushFreq=2000,
>> walFsyncDelay=1000, walRecordIterBuffSize=67108864,
>> alwaysWriteFullPages=false,
>> fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@4bd31064,
>> metricsSubIntervalCnt=5, metricsRateTimeInterval=60000,
>> walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false,
>> walCompactionEnabled=false], activeOnStart=true, autoActivation=true,
>> longQryWarnTimeout=3000, sqlConnCfg=null,
>> cliConnCfg=ClientConnectorConfiguration [host=null, port=10800,
>> portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true,
>> maxOpenCursorsPerConn=128, threadPoolSize=32, idleTimeout=0,
>> jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true,
>> sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false,
>> sslCtxFactory=null], authEnabled=false, failureHnd=null,
>> commFailureRslvr=null]
>>
>>
>>
>> and error while connecting client:
>>
>> [2020-04-14T09:41:33,547][WARN
>> ][grid-timeout-worker-#71%igniteStart%][TcpDiscoverySpi] Socket write
>> has timed out (consider increasing 'sockTimeout' configuration
>> property) [sockTimeout=5000, rmtAddr=/10.80.104.224:51856,
>> rmtPort=51856, sockTimeout=5000]
>>
>> In server configuration we didn't define any socketTimeout, server
>> might be throwing socket timeout not client. But It occurs for only
>> one particular client and this server. Other web applications are able
>> to connect with same server on our production environment.
>>
>> Thanks
>>
>> On Mon, Apr 13, 2020 at 8:09 PM Evgenii Zhuravlev
>> <e.zhuravlev.wk@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > Can you share full logs from all nodes? I mean log files, not the console output.
>> >
>> > Evgenii
>> >
>> > вс, 12 апр. 2020 г. в 20:30, Rajan Ahlawat <rajan.ahlawat@gmail.com>:
>> >>
>> >> ?
>> >>
>> >> On Thu, Apr 9, 2020 at 3:11 AM Rajan Ahlawat <rajan.ahlawat@gmail.com> wrote:
>> >> >
>> >> > ---------- Forwarded message ---------
>> >> > From: Rajan Ahlawat <rajan.ahlawat@gmail.com>
>> >> > Date: Thu, Apr 9, 2020 at 3:09 AM
>> >> > Subject: org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - Failed
>> >> > to reconnect to cluster (will retry): class
>> >> > o.a.i.IgniteCheckedException: Failed to deserialize object with given
>> >> > class loader: org.springframework.boot.loader.LaunchedURLClassLoader
>> >> > To: <user@ignite.apache.org>
>> >> >
>> >> >
>> >> > Hi
>> >> >
>> >> > We suddenly started getting following exception on client side after
>> >> > node running application got restarted:
>> >> >
>> >> > org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - Failed to
>> >> > reconnect to cluster (will retry): class o.a.i.IgniteCheckedException:
>> >> > Failed to deserialize object with given class loader:
>> >> > org.springframework.boot.loader.LaunchedURLClassLoader
>> >> >
>> >> > I see similar bug was raised here for version 2.7.0:
>> >> > https://issues.apache.org/jira/browse/IGNITE-11730
>> >> >
>> >> > We are currently using version 2.6.0
>> >> > Following is our tcpDiscoveryApi configurations:
>> >> >
>> >> > private void setDiscoverySpiConfig(IgniteConfiguration cfg) {
>> >> >     TcpDiscoverySpi discoverySpi = new TcpDiscoverySpi();
>> >> >
>> >> >     setIpFinder(discoverySpi);
>> >> >     discoverySpi.setNetworkTimeout(platformCachingConfiguration.getIgnite().getSocketTimeout());
>> >> >     discoverySpi.setSocketTimeout(platformCachingConfiguration.getIgnite().getSocketTimeout());
>> >> >     discoverySpi.setJoinTimeout(platformCachingConfiguration.getIgnite().getJoinTimeout());
>> >> >     discoverySpi.setClientReconnectDisabled(platformCachingConfiguration.getIgnite().isClientReconnectDisabled());
>> >> >     discoverySpi.setReconnectCount(platformCachingConfiguration.getIgnite().getReconnectCount());
>> >> >     discoverySpi.setReconnectDelay(platformCachingConfiguration.getIgnite().getReconnectDelay());
>> >> >
>> >> >     cfg.setDiscoverySpi(discoverySpi);
>> >> > }
>> >> >
>> >> > Its IPfinder config is
>> >> >
>> >> > private void setTcpIpFinder(TcpDiscoverySpi discoverySpi) {
>> >> >     TcpDiscoveryVmIpFinder ipFinder = new TcpDiscoveryVmIpFinder();
>> >> >
>> >> >     ipFinder.setAddresses(platformCachingConfiguration.getIgnite().getNodes());
>> >> >     discoverySpi.setIpFinder(ipFinder);
>> >> > }
>> >> >
>> >> > We have tried every combination of timeouts, right now timeouts are
>> >> > set at very hight value .
>> >> >
>> >> > (1) If we are having same bug mentioned for 2.7.0 version, but bug
>> >> > desc says it occurs on server side, but we are getting exact same
>> >> > stack trance in ClientImpl.java on client side.
>> >> > (2) assuming it is same issues, is there a way to disable data bag
>> >> > compression check, since upgrading both client and server version
>> >> > would not be possible immediately.
>> >> >
>> >> > Thanks in advance.