ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergey Chugunov (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-9673) Timeout in Java Client suite.
Date Wed, 24 Oct 2018 10:49:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-9673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662100#comment-16662100
] 

Sergey Chugunov commented on IGNITE-9673:
-----------------------------------------

[~NSAmelchev],

I reviewed the change and the overall idea of moving *makeResponse* call to system pool looks
reasonable to me.

At the same time I see that the change touches a lot of handlers that are not affected by
deadlock. Are there any risks of moving them to system pool? [~ilantukh], what do you think
on this?

> Timeout in Java Client suite.
> -----------------------------
>
>                 Key: IGNITE-9673
>                 URL: https://issues.apache.org/jira/browse/IGNITE-9673
>             Project: Ignite
>          Issue Type: Bug
>            Reporter: Amelchev Nikita
>            Assignee: Amelchev Nikita
>            Priority: Major
>              Labels: MakeTeamcityGreenAgain
>             Fix For: 2.8
>
>         Attachments: ThreadDump.txt
>
>
> Example of timeout: [TC build|[https://ci.ignite.apache.org/viewLog.html?buildId=1919405&tab=buildResultsDiv&buildTypeId=IgniteTests24Java8_JavaClient].]
> The possible reason is non-interruptable future and starvation in stripped pool:
> {noformat}
> "test-runner-#2440%redis.RedisProtocolStringSelfTest%" #3843 prio=5 os_prio=0 tid=0x00007f8f053fb000
nid=0x7b19 waiting on condition [0x00007f8d74f8f000]
>    java.lang.Thread.State: WAITING (parking)
> 	at sun.misc.Unsafe.park(Native Method)
> 	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> 	at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
> 	at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
> 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2465)
> 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$22.op(GridCacheAdapter.java:2463)
> 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.syncOp(GridCacheAdapter.java:4228)
> 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put0(GridCacheAdapter.java:2463)
> 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2444)
> 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter.put(GridCacheAdapter.java:2421)
> 	at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.put(IgniteCacheProxyImpl.java:1089)
> 	at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.put(GatewayProtectedCacheProxy.java:820)
> 	at org.apache.ignite.internal.processors.rest.protocols.tcp.redis.RedisProtocolStringSelfTest.testStrlen(RedisProtocolStringSelfTest.java:310)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at junit.framework.TestCase.runTest(TestCase.java:176)
> 	at org.apache.ignite.testframework.junits.GridAbstractTest.runTestInternal(GridAbstractTest.java:2177)
> 	at org.apache.ignite.testframework.junits.GridAbstractTest.access$000(GridAbstractTest.java:143)
> 	at org.apache.ignite.testframework.junits.GridAbstractTest$5.run(GridAbstractTest.java:2092)
> 	at java.lang.Thread.run(Thread.java:748)
> [grid-timeout-worker-#2323%redis.RedisProtocolStringSelfTest0%][G] >>> Possible
starvation in striped pool.
> Thread name: sys-stripe-3-#2304%redis.RedisProtocolStringSelfTest0%
> Queue: [Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false,
timeout=0, skipOnTimeout=false, msg=GridCacheIdMessage [cacheId=1481046058]GridDistributedBaseMessage
[ver=GridCacheVersion [topVer=148979816, order=1537499815759, nodeOrder=1], committedVers=ArrayList
[], rolledbackVers=ArrayList [], cnt=0, super=]GridDistributedLockResponse [futId=e739f1af561-9bc10183-74c7-4b9a-a525-aef32c002efc,
err=null, vals=ArrayList [null], super=]GridNearLockResponse [pending=ArrayList [], miniId=1,
dhtVers=GridCacheVersion[] [GridCacheVersion [topVer=0, order=0, nodeOrder=0]], mappedVers=GridCacheVersion[]
[GridCacheVersion [topVer=148979816, order=1537499815760, nodeOrder=2]], clientRemapVer=null,
super=]]], Message closure [msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false,
timeout=0, skipOnTimeout=false, msg=GridDistributedTxFinishResponse [txId=GridCacheVersion
[topVer=148979816, order=1537499815765, nodeOrder=1], futId=e849f1af561-9bc10183-74c7-4b9a-a525-aef32c002efc,
part=-1]GridNearTxFinishResponse [err=null, miniId=1, nearThreadId=3843, super=]]]]
> Deadlock: false
> Completed: 5
> "sys-stripe-3-#2304%redis.RedisProtocolStringSelfTest0%" #3628 prio=5 os_prio=0 tid=0x00007f8f054e1800
nid=0x7a41 waiting on condition [0x00007f8ed2ef0000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:304)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:177)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:140)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4715)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4689)
> at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1413)
> at org.apache.ignite.internal.processors.cache.GridCacheProxyImpl.get(GridCacheProxyImpl.java:332)
> at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$4.applyx(DataStructuresProcessor.java:562)
> at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor$4.applyx(DataStructuresProcessor.java:557)
> at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.retryTopologySafe(DataStructuresProcessor.java:1653)
> at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.getAtomic(DataStructuresProcessor.java:557)
> at org.apache.ignite.internal.processors.datastructures.DataStructuresProcessor.atomicLong(DataStructuresProcessor.java:463)
> at org.apache.ignite.internal.IgniteKernal.atomicLong(IgniteKernal.java:3569)
> at org.apache.ignite.internal.IgniteKernal.atomicLong(IgniteKernal.java:3558)
> at org.apache.ignite.internal.processors.rest.handlers.redis.string.GridRedisGetCommandHandler.makeResponse(GridRedisGetCommandHandler.java:90)
> at org.apache.ignite.internal.processors.rest.handlers.redis.GridRedisRestCommandHandler$1.applyx(GridRedisRestCommandHandler.java:72)
> at org.apache.ignite.internal.processors.rest.handlers.redis.GridRedisRestCommandHandler$1.applyx(GridRedisRestCommandHandler.java:66)
> at org.apache.ignite.internal.util.lang.IgniteClosureX.apply(IgniteClosureX.java:38)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:179)
> at org.apache.ignite.internal.processors.rest.GridRestProcessor$2$1.apply(GridRestProcessor.java:176)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.applyCallback(GridFutureChainListener.java:78)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:70)
> at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:30)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at org.apache.ignite.internal.processors.cache.GridCacheCompoundIdentityFuture.onDone(GridCacheCompoundIdentityFuture.java:56)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture.onDone(GridNearGetFuture.java:220)
> at org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture.onDone(GridNearGetFuture.java:69)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:285)
> at org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:144)
> at org.apache.ignite.internal.util.future.GridCompoundFuture.apply(GridCompoundFuture.java:45)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:383)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblock(GridFutureAdapter.java:347)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.unblockAll(GridFutureAdapter.java:335)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:495)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:474)
> at org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture$MiniFuture.onDone(GridNearGetFuture.java:914)
> at org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture$MiniFuture.onDone(GridNearGetFuture.java:837)
> at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:451)
> at org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture$MiniFuture.postProcessResultAndDone(GridNearGetFuture.java:1041)
> at org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture$MiniFuture.onResult(GridNearGetFuture.java:1027)
> at org.apache.ignite.internal.processors.cache.distributed.near.GridNearGetFuture.onResult(GridNearGetFuture.java:213)
> at org.apache.ignite.internal.processors.cache.distributed.near.GridNearCacheAdapter.processGetResponse(GridNearCacheAdapter.java:291)
> at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTransactionalCache$1.apply(GridNearTransactionalCache.java:92)
> at org.apache.ignite.internal.processors.cache.distributed.near.GridNearTransactionalCache$1.apply(GridNearTransactionalCache.java:90)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1056)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:581)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:380)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:306)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:101)
> at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:295)
> at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1569)
> at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1197)
> at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:127)
> at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1093)
> at org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:496)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
> The main thread is waiting for runner thread interrupt:
> {noformat}
> "main" #1 prio=5 os_prio=0 tid=0x00007f8f0400e000 nid=0x6c18 in Object.wait() [0x00007f8f0db58000]
>    java.lang.Thread.State: WAITING (on object monitor)
> 	at java.lang.Object.wait(Native Method)
> 	at java.lang.Thread.join(Thread.java:1252)
> 	- locked <0x000000078bfaf668> (a org.apache.ignite.thread.IgniteThread)
> 	at org.apache.ignite.internal.util.IgniteUtils.join(IgniteUtils.java:4662)
> 	at org.apache.ignite.internal.util.IgniteUtils.join(IgniteUtils.java:4647)
> 	at org.apache.ignite.testframework.junits.GridAbstractTest.runTest(GridAbstractTest.java:2124)
> 	at junit.framework.TestCase.runBare(TestCase.java:141)
> 	at junit.framework.TestResult$1.protect(TestResult.java:122)
> 	at junit.framework.TestResult.runProtected(TestResult.java:142)
> 	at junit.framework.TestResult.run(TestResult.java:125)
> 	at junit.framework.TestCase.run(TestCase.java:129)
> 	at junit.framework.TestSuite.runTest(TestSuite.java:255)
> 	at junit.framework.TestSuite.run(TestSuite.java:250)
> 	at junit.framework.TestSuite.runTest(TestSuite.java:255)
> 	at junit.framework.TestSuite.run(TestSuite.java:250)
> 	at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:84)
> 	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:369)
> 	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:275)
> 	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:239)
> 	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:160)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:206)
> 	at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:160)
> 	at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:83)
> 	at org.apache.maven.plugin.surefire.InPluginVMSurefireStarter.runSuitesInProcess(InPluginVMSurefireStarter.java:84)
> 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1107)
> 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:954)
> 	at org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:832)
> 	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:137)
> 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:154)
> 	at org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:146)
> 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:117)
> 	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:81)
> 	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:56)
> 	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:128)
> 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:305)
> 	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:192)
> 	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:105)
> 	at org.apache.maven.cli.MavenCli.execute(MavenCli.java:956)
> 	at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:290)
> 	at org.apache.maven.cli.MavenCli.main(MavenCli.java:194)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:498)
> 	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> 	at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> 	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> 	at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> {noformat}
> It happens in the GridAbstractTest.runTest() method and prevents test interrupt by timeout:
> {noformat}
>             U.interrupt(runner);
>             U.join(runner, log);
>             throw new TimeoutException("Test has been timed out [test=" + getName() +
", timeout=" +
>                 getTestTimeout() + ']' );
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message