ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ilya Suntsov (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (IGNITE-732) IgniteCache.size() should not fail in case of topology changes.
Date Mon, 25 May 2015 12:23:17 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558163#comment-14558163
] 

Ilya Suntsov edited comment on IGNITE-732 at 5/25/15 12:22 PM:
---------------------------------------------------------------

Steps for reproduction:
1. Start 3 nodes with ExampleNodeStartup
2. Start StreamWords
3. Start QueryWords
I observed the following behavior:
{noformat}
[14:26:26]    __________  ________________ 
[14:26:26]   /  _/ ___/ |/ /  _/_  __/ __/ 
[14:26:26]  _/ // (7 7    // /  / / / _/   
[14:26:26] /___/\___/_/|_/___/ /_/ /___/  
[14:26:26]  
[14:26:26] ver. 1.0.7-QAIS2501#20150522-sha1:050f4292
[14:26:26] 2015 Copyright(C) Apache Software Foundation
[14:26:26] 
[14:26:26] Quiet mode.
[14:26:26]   ^-- Logging to file '/Users/gridgain/Downloads/gridgain-enterprise-fabric-7.0.7-QAIS2501/work/log/ignite-364df547.log'
[14:26:26]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[14:26:26] 
[14:26:26] Initial heap size is 192MB (should be no less than 512MB, use -Xms512m -Xmx512m).
[14:26:26] Configured plugins:
[14:26:26]   ^-- GridGain 7.0.7-QAIS2501#20150525-sha1:e7f977ac
[14:26:26]   ^-- 2015 Copyright(C) GridGain Systems
[14:26:26] 
[14:26:30] Attempting to start more nodes than physical RAM available on current host (this
can cause significant slowdown)
[14:26:32,975][ERROR][main][GridEntLicenseProcessor] License bursting ==> Maximum number
of nodes (2) is exceeded.
[14:26:32,975][ERROR][main][GridEntLicenseProcessor]   ^-- Contact sales@gridgain.com for
further assistance. Make sure to include your license ID: 14177F1A-5B5A-4F9C-BC55-F7335715F311
[14:26:32,976][ERROR][main][GridEntLicenseProcessor] License grace/burst period - left 1 hour.
[14:26:32] Performance suggestions for grid  (fix if possible)
[14:26:32] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[14:26:32]   ^-- Disable peer class loading (set 'peerClassLoadingEnabled' to false)
[14:26:32]   ^-- Disable grid events (remove 'includeEventTypes' from configuration)
[14:26:32] 
[14:26:32] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[14:26:32] 
[14:26:32] Ignite node started OK (id=364df547)
[14:26:33] Topology snapshot [ver=5, nodes=5, CPUs=8, heap=13.0GB]
[14:26:36] New version is available at http://www.gridgain.com/download/editions/#enterprise-edition:
7.0.5
Query results [avg=4.81, min=1, max=1730]
(in,673)
(it,585)
(to,526)
(a,431)
(I,422)
(that,371)
(she,337)
(said,288)
(or,246)
(for,245)
Query results [avg=4.50, min=1, max=429]
(the,2230)
(and,1065)
(of,831)
(to,796)
(a,662)
(she,535)
(said,424)
(was,347)
(Alice,276)
(you,273)
Query results [avg=2.95, min=1, max=266]
...
(the,3219)
(to,1681)
(and,1549)
(a,1474)
(of,1248)
(she,1047)
(said,959)
(in,876)
(it,801)
(was,714)
[14:27:22] Topology snapshot [ver=6, nodes=4, CPUs=8, heap=11.0GB]
[14:27:22] Ignite node stopped OK [uptime=00:00:49:513]
Exception in thread "main" javax.cache.CacheException: Failed to run map query remotely.
	at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:343)
	at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:775)
	at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:866)
	at org.apache.ignite.internal.processors.query.GridQueryProcessor.queryTwoStep(GridQueryProcessor.java:561)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:513)
	at org.apache.ignite.examples.streaming.wordcount.QueryWords.main(QueryWords.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: javax.cache.CacheException: Failed to execute map query on the node: 30a59fd5-ac86-46a6-9c01-1307260a0e6c,
Node left the topology.
	at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.fail(GridReduceQueryExecutor.java:204)
	at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.access$300(GridReduceQueryExecutor.java:61)
	at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor$2.onEvent(GridReduceQueryExecutor.java:147)
	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:745)
	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:730)
	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:270)
	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1597)
	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1787)
	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1636)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
	at java.lang.Thread.run(Thread.java:745)
{noformat}

Is it correct or not?



was (Author: ustas):
Steps for reproduction:
1. Start 3 nodes with ExampleNodeStartup
2. Start StreamWords
3. Start QueryWords
I observed the following behavior:
{noformat}
[14:26:26]    __________  ________________ 
[14:26:26]   /  _/ ___/ |/ /  _/_  __/ __/ 
[14:26:26]  _/ // (7 7    // /  / / / _/   
[14:26:26] /___/\___/_/|_/___/ /_/ /___/  
[14:26:26]  
[14:26:26] ver. 1.0.7-QAIS2501#20150522-sha1:050f4292
[14:26:26] 2015 Copyright(C) Apache Software Foundation
[14:26:26] 
[14:26:26] Quiet mode.
[14:26:26]   ^-- Logging to file '/Users/gridgain/Downloads/gridgain-enterprise-fabric-7.0.7-QAIS2501/work/log/ignite-364df547.log'
[14:26:26]   ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat}
[14:26:26] 
[14:26:26] Initial heap size is 192MB (should be no less than 512MB, use -Xms512m -Xmx512m).
[14:26:26] Configured plugins:
[14:26:26]   ^-- GridGain 7.0.7-QAIS2501#20150525-sha1:e7f977ac
[14:26:26]   ^-- 2015 Copyright(C) GridGain Systems
[14:26:26] 
[14:26:30] Attempting to start more nodes than physical RAM available on current host (this
can cause significant slowdown)
[14:26:32,975][ERROR][main][GridEntLicenseProcessor] License bursting ==> Maximum number
of nodes (2) is exceeded.
[14:26:32,975][ERROR][main][GridEntLicenseProcessor]   ^-- Contact sales@gridgain.com for
further assistance. Make sure to include your license ID: 14177F1A-5B5A-4F9C-BC55-F7335715F311
[14:26:32,976][ERROR][main][GridEntLicenseProcessor] License grace/burst period - left 1 hour.
[14:26:32] Performance suggestions for grid  (fix if possible)
[14:26:32] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[14:26:32]   ^-- Disable peer class loading (set 'peerClassLoadingEnabled' to false)
[14:26:32]   ^-- Disable grid events (remove 'includeEventTypes' from configuration)
[14:26:32] 
[14:26:32] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[14:26:32] 
[14:26:32] Ignite node started OK (id=364df547)
[14:26:33] Topology snapshot [ver=5, nodes=5, CPUs=8, heap=13.0GB]
[14:26:36] New version is available at http://www.gridgain.com/download/editions/#enterprise-edition:
7.0.5
Query results [avg=4.81, min=1, max=1730]
(in,673)
(it,585)
(to,526)
(a,431)
(I,422)
(that,371)
(she,337)
(said,288)
(or,246)
(for,245)
Query results [avg=4.50, min=1, max=429]
(the,2230)
(and,1065)
(of,831)
(to,796)
(a,662)
(she,535)
(said,424)
(was,347)
(Alice,276)
(you,273)
Query results [avg=2.95, min=1, max=266]
...
(the,3219)
(to,1681)
(and,1549)
(a,1474)
(of,1248)
(she,1047)
(said,959)
(in,876)
(it,801)
(was,714)
[14:27:22] Topology snapshot [ver=6, nodes=4, CPUs=8, heap=11.0GB]
[14:27:22] Ignite node stopped OK [uptime=00:00:49:513]
Exception in thread "main" javax.cache.CacheException: Failed to run map query remotely.
	at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:343)
	at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:775)
	at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:866)
	at org.apache.ignite.internal.processors.query.GridQueryProcessor.queryTwoStep(GridQueryProcessor.java:561)
	at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:513)
	at org.apache.ignite.examples.streaming.wordcount.QueryWords.main(QueryWords.java:67)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: javax.cache.CacheException: Failed to execute map query on the node: 30a59fd5-ac86-46a6-9c01-1307260a0e6c,
Node left the topology.
	at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.fail(GridReduceQueryExecutor.java:204)
	at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.access$300(GridReduceQueryExecutor.java:61)
	at org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor$2.onEvent(GridReduceQueryExecutor.java:147)
	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:745)
	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:730)
	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:270)
	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1597)
	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1787)
	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1636)
	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
	at java.lang.Thread.run(Thread.java:745)
{noformat}

Is it correct or not?


> IgniteCache.size() should not fail in case of topology changes.
> ---------------------------------------------------------------
>
>                 Key: IGNITE-732
>                 URL: https://issues.apache.org/jira/browse/IGNITE-732
>             Project: Ignite
>          Issue Type: Bug
>          Components: cache
>            Reporter: Dmitriy Setrakyan
>            Assignee: Yakov Zhdanov
>            Priority: Blocker
>             Fix For: sprint-5
>
>
> I believe that cache.size() method fails if topology changes during the execution. I
got the following error when executing word-count {{QueryWords}} class and removing one of
the server nodes.
> {code}
> Query results [total=1003726, avg=4.34, min=1, max=1314]
> (the,20)
> (and,9)
> (a,8)
> (in,8)
> (of,8)
> (that,5)
> (said,4)
> (on,4)
> (for,4)
> (to,4)
> [17:35:54] Topology snapshot [ver=5, nodes=3, CPUs=4, heap=4.0GB]
> [17:35:54,772][ERROR][disco-event-worker-#46%null%][GridTaskWorker] Failed to obtain
remote job result policy for result from ComputeTask.result(..) method (will fail the whole
task): GridJobResultImpl [job=C1 [arg=null], sib=GridJobSiblingImpl [sesId=9a42d9fac41-8437ac61-27ef-4e4b-951d-a29b18346bfe,
jobId=ba42d9fac41-833cae7a-295f-4578-a6b0-f4b6d3a1bf50, nodeId=833cae7a-295f-4578-a6b0-f4b6d3a1bf50,
isJobDone=false], jobCtx=GridJobContextImpl [jobId=ba42d9fac41-833cae7a-295f-4578-a6b0-f4b6d3a1bf50,
attrs={}], node=TcpDiscoveryNode [id=833cae7a-295f-4578-a6b0-f4b6d3a1bf50, addrs=[192.168.2.10,
0:0:0:0:0:0:0:1, 127.0.0.1], sockAddrs=[/192.168.2.10:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500],
discPort=47500, order=1, intOrder=1, loc=false, ver=1.0.0#19691231-sha1:00000000, isClient=false],
ex=class o.a.i.cluster.ClusterTopologyException: Node has left grid: 833cae7a-295f-4578-a6b0-f4b6d3a1bf50,
hasRes=true, isCancelled=false, isOccupied=true]
> class org.apache.ignite.cluster.ClusterTopologyException: Node has left grid: 833cae7a-295f-4578-a6b0-f4b6d3a1bf50
> 	at org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1204)
> 	at org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener.onEvent(GridTaskProcessor.java:1180)
> 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:739)
> 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:724)
> 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:272)
> 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1533)
> 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1720)
> 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1572)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
> 	at java.lang.Thread.run(Thread.java:745)
> [17:35:55] Ignite node stopped OK [uptime=02:45:01:791]
> Exception in thread "main" javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException:
Node has left grid: 833cae7a-295f-4578-a6b0-f4b6d3a1bf50
> 	at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1808)
> 	at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1463)
> 	at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.size(IgniteCacheProxy.java:594)
> 	at org.apache.ignite.examples.streaming.wordcount.QueryWords.main(QueryWords.java:69)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:483)
> 	at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> Caused by: class org.apache.ignite.IgniteCheckedException: Node has left grid: 833cae7a-295f-4578-a6b0-f4b6d3a1bf50
> 	at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:6715)
> 	at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:124)
> 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$68.applyx(GridCacheAdapter.java:3757)
> 	at org.apache.ignite.internal.processors.cache.GridCacheAdapter$68.applyx(GridCacheAdapter.java:3754)
> 	at org.apache.ignite.internal.util.lang.IgniteClosureX.apply(IgniteClosureX.java:37)
> 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:54)
> 	at org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:28)
> 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:241)
> 	at org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:229)
> 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:359)
> 	at org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:325)
> 	at org.apache.ignite.internal.processors.task.GridTaskWorker.finishTask(GridTaskWorker.java:1323)
> 	at org.apache.ignite.internal.processors.task.GridTaskWorker.finishTask(GridTaskWorker.java:1291)
> 	at org.apache.ignite.internal.processors.task.GridTaskWorker$3.apply(GridTaskWorker.java:871)
> 	at org.apache.ignite.internal.processors.task.GridTaskWorker$3.apply(GridTaskWorker.java:822)
> 	at org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6139)
> 	at org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:822)
> 	at org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:718)
> 	at org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1223)
> 	at org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener.onEvent(GridTaskProcessor.java:1180)
> 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:739)
> 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:724)
> 	at org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:272)
> 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1533)
> 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1720)
> 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1572)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
> 	at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.cluster.ClusterTopologyException: Node has left grid:
833cae7a-295f-4578-a6b0-f4b6d3a1bf50
> 	at org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1204)
> 	... 9 more
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message