ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dmitriy Setrakyan (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-3211) "Failed to reinitialize local partitions", "Failed to wait for completion of partition map exchange" errors during failover test
Date Mon, 30 May 2016 15:41:12 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Dmitriy Setrakyan updated IGNITE-3211:
--------------------------------------
    Fix Version/s: 1.7

> "Failed to reinitialize local partitions", "Failed to wait for completion of partition
map exchange" errors during failover test
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: IGNITE-3211
>                 URL: https://issues.apache.org/jira/browse/IGNITE-3211
>             Project: Ignite
>          Issue Type: Bug
>    Affects Versions: 1.6
>            Reporter: Ksenia Rybakova
>             Fix For: 1.7
>
>
> "Failed to reinitialize local partitions (preloading will be stopped)" and "Failed to
wait for completion of partition map exchange (preloading will not start)" errors occured
during failover load test. Complete stack trace see below.
> Load config: 
> - 1 client, 20 servers (5 servers per 1 host)
> - warmup 60 
> - duration 66h
> - preload 5M
> - key range 10M
> - operations: PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL PUT_IF_ABSENT
REPLACE
> - backups count 3
> - 3 servers restart every 15 min with 30 sec step, pause between stop and start 5min
> {noformat}
> [08:32:21,002][ERROR][exchange-worker-#83%null%][GridDhtPartitionsExchangeFuture] Failed
to reinitialize local partitions (preloading will be stopped): GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=39, minorTopVer=1], nodeId=20ddc8b7, evt=DISCOVERY_CUSTOM_EVT]
> class org.apache.ignite.IgniteException: null
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:506)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:297)
>         at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:3743)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.toString(GridDhtTxLocalAdapter.java:868)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.toString(GridDhtTxLocal.java:703)
>         at java.lang.String.valueOf(String.java:2849)
>         at java.lang.StringBuilder.append(StringBuilder.java:128)
>         at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpPendingObjects(GridCachePartitionExchangeManager.java:1172)
>         at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpDebugInfo(GridCachePartitionExchangeManager.java:1150)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.dumpPendingObjects(GridDhtPartitionsExchangeFuture.java:894)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:769)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:715)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:472)
>         at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1333)
>         at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.IgniteException: null
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:506)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:364)
>         at org.apache.ignite.internal.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:443)
>         at java.lang.String.valueOf(String.java:2849)
>         at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
>         ... 15 more
> Caused by: java.util.ConcurrentModificationException
>         at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
>         at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413)
>         at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412)
>         at java.util.AbstractMap.toString(AbstractMap.java:518)
>         at java.lang.String.valueOf(String.java:2849)
>         at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
>         ... 20 more
> [08:32:21,072][ERROR][exchange-worker-#83%null%][GridCachePartitionExchangeManager] Failed
to wait for completion of partition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture
[dummy=false, forcePreload=false, reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null,
affTopVer=AffinityTopologyVersion [topVer=39, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47500,
/10.20.0.216:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1464363141003,
loc=true, ver=1.6.0#20160525-sha1:48321a40, isClient=false], topVer=39, nodeId8=20ddc8b7,
msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1464363130370]], crd=TcpDiscoveryNode [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419,
addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:47500,
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1464363141003, loc=true,
ver=1.6.0#20160525-sha1:48321a40, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=39, minorTopVer=1], nodeId=20ddc8b7, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter
[resFlag=2, res=false, startTime=1464363130370, endTime=1464363141057, ignoreInterrupts=false,
state=DONE], init=false, topSnapshot=null, lastVer=GridCacheVersion [topVer=75840698, time=1464363140466,
order=1464462577173, nodeOrder=38], partReleaseFut=GridCompoundFuture [rdc=null, initFlag=1,
lsnrCalls=2, done=false, cancelled=false, err=null, futs=[true, false, true]], affChangeMsg=CacheAffinityChangeMessage
[id=9560ad2f451-621c2001-9d60-40f4-ab21-20834a2b6b33, topVer=AffinityTopologyVersion [topVer=39,
minorTopVer=0], exchId=null, partsMsg=null, exchangeNeeded=true], skipPreload=false, clientOnlyExchange=false,
initTs=1464363130370, centralizedAff=false, evtLatch=0, remaining=[e7f032f5-4ec6-48c8-9cd4-ac78b0f8ccc4,
a550929c-1f4b-4de3-8d57-47b4a8232010, c7ff42e7-acd5-4cb4-9cd5-179aa18b88b0, fc79bff1-480f-44b3-9915-2548f937726c,
c013837c-b46e-4667-9e47-e3c41412b5a1, ba89970d-8908-4276-9d3c-b679a335c2d2, a7f0feb2-b932-4843-ac93-560fc547700e,
3c462bba-1d12-43ab-ad21-761f9ef94aa1, 9c184df6-e5e0-49b3-9836-65932d58be6a, 317b61fc-186b-4bf7-9622-6c4094409814,
425b461c-5117-403d-af2b-e323b7c1aa39, a4d0911c-d54a-4a36-b911-b74eddc3f0cc, d0e84363-590d-45bf-bad3-b88cde5d03fe,
b51cc458-c884-41ce-9e5b-b83ea111218b, 6970f13f-fe82-4583-a128-af93f86e99e7, a9b8fe06-cda9-4907-871d-0684bcc2c3ca,
c978e55f-4de6-4ffd-98b1-459215db2232, cab5d0e0-7365-4774-8f99-d9f131c5d896, 0859feda-b70d-4fab-870a-4db18bd5984b],
srvNodes=[TcpDiscoveryNode [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, addrs=[10.20.0.216, 127.0.0.1],
sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:47500, /127.0.0.1:47500], discPort=47500,
order=1, intOrder=1, lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=cab5d0e0-7365-4774-8f99-d9f131c5d896, addrs=[10.20.0.221,
127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47500, /10.20.0.221:47500, /127.0.0.1:47500],
discPort=47500, order=2, intOrder=2, lastExchangeTime=1464360659615, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=c7ff42e7-acd5-4cb4-9cd5-179aa18b88b0, addrs=[10.20.0.222,
127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47500, /10.20.0.222:47500, /127.0.0.1:47500],
discPort=47500, order=3, intOrder=3, lastExchangeTime=1464360660162, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=0859feda-b70d-4fab-870a-4db18bd5984b, addrs=[10.20.0.223,
127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47500, /10.20.0.223:47500, /127.0.0.1:47500],
discPort=47500, order=4, intOrder=4, lastExchangeTime=1464360660689, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=ba89970d-8908-4276-9d3c-b679a335c2d2, addrs=[10.20.0.216,
127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47501, /10.20.0.216:47501, /127.0.0.1:47501],
discPort=47501, order=5, intOrder=5, lastExchangeTime=1464360661276, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=317b61fc-186b-4bf7-9622-6c4094409814, addrs=[10.20.0.221,
127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47501, /10.20.0.221:47501, /127.0.0.1:47501],
discPort=47501, order=6, intOrder=6, lastExchangeTime=1464360662086, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=425b461c-5117-403d-af2b-e323b7c1aa39, addrs=[10.20.0.222,
127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47501, /10.20.0.222:47501, /127.0.0.1:47501],
discPort=47501, order=7, intOrder=7, lastExchangeTime=1464360662471, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=a550929c-1f4b-4de3-8d57-47b4a8232010, addrs=[10.20.0.223,
127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47501, /10.20.0.223:47501, /127.0.0.1:47501],
discPort=47501, order=8, intOrder=8, lastExchangeTime=1464360662886, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=e7f032f5-4ec6-48c8-9cd4-ac78b0f8ccc4, addrs=[10.20.0.216,
127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47502, /10.20.0.216:47502, /127.0.0.1:47502],
discPort=47502, order=9, intOrder=9, lastExchangeTime=1464360663373, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=b51cc458-c884-41ce-9e5b-b83ea111218b, addrs=[10.20.0.221,
127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47502, /10.20.0.221:47502, /127.0.0.1:47502],
discPort=47502, order=10, intOrder=10, lastExchangeTime=1464360664058, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=fc79bff1-480f-44b3-9915-2548f937726c, addrs=[10.20.0.222,
127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47502, /10.20.0.222:47502, /127.0.0.1:47502],
discPort=47502, order=11, intOrder=11, lastExchangeTime=1464360664685, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=a7f0feb2-b932-4843-ac93-560fc547700e, addrs=[10.20.0.223,
127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47502, /10.20.0.223:47502, /127.0.0.1:47502],
discPort=47502, order=12, intOrder=12, lastExchangeTime=1464360665216, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=9c184df6-e5e0-49b3-9836-65932d58be6a, addrs=[10.20.0.216,
127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47503, /10.20.0.216:47503, /127.0.0.1:47503],
discPort=47503, order=13, intOrder=13, lastExchangeTime=1464360665661, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=6970f13f-fe82-4583-a128-af93f86e99e7, addrs=[10.20.0.216,
127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47504, /10.20.0.216:47504, /127.0.0.1:47504],
discPort=47504, order=17, intOrder=17, lastExchangeTime=1464360668158, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=d0e84363-590d-45bf-bad3-b88cde5d03fe, addrs=[10.20.0.221,
127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47504, /10.20.0.221:47504, /127.0.0.1:47504],
discPort=47504, order=18, intOrder=18, lastExchangeTime=1464360668768, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=c013837c-b46e-4667-9e47-e3c41412b5a1, addrs=[10.20.0.222,
127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47504, /10.20.0.222:47504, /127.0.0.1:47504],
discPort=47504, order=19, intOrder=19, lastExchangeTime=1464360669745, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=3c462bba-1d12-43ab-ad21-761f9ef94aa1, addrs=[10.20.0.223,
127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47504, /10.20.0.223:47504, /127.0.0.1:47504],
discPort=47504, order=20, intOrder=20, lastExchangeTime=1464360669836, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=c978e55f-4de6-4ffd-98b1-459215db2232, addrs=[10.20.0.221,
127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47503, /10.20.0.221:47503, /127.0.0.1:47503],
discPort=47503, order=37, intOrder=28, lastExchangeTime=1464362949633, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=a9b8fe06-cda9-4907-871d-0684bcc2c3ca, addrs=[10.20.0.222,
127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47503, /10.20.0.222:47503, /127.0.0.1:47503],
discPort=47503, order=38, intOrder=29, lastExchangeTime=1464362981162, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=a4d0911c-d54a-4a36-b911-b74eddc3f0cc, addrs=[10.20.0.223,
127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47503, /10.20.0.223:47503, /127.0.0.1:47503],
discPort=47503, order=39, intOrder=30, lastExchangeTime=1464363011016, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false]], super=GridFutureAdapter [resFlag=1, res=class o.a.i.IgniteException: null,
startTime=1464363130370, endTime=1464363141003, ignoreInterrupts=false, state=DONE]]
> class org.apache.ignite.IgniteCheckedException: null
>         at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7067)
>         at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:168)
>         at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:117)
>         at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1335)
>         at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
>         at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
>         at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
>         at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413)
>         at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412)
>         at java.util.AbstractMap.toString(AbstractMap.java:518)
>         at java.lang.String.valueOf(String.java:2849)
>         at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:364)
>         at org.apache.ignite.internal.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:443)
>         at java.lang.String.valueOf(String.java:2849)
>         at org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
>         at org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:297)
>         at org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:3743)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.toString(GridDhtTxLocalAdapter.java:868)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.toString(GridDhtTxLocal.java:703)
>         at java.lang.String.valueOf(String.java:2849)
>         at java.lang.StringBuilder.append(StringBuilder.java:128)
>         at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpPendingObjects(GridCachePartitionExchangeManager.java:1172)
>         at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpDebugInfo(GridCachePartitionExchangeManager.java:1150)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.dumpPendingObjects(GridDhtPartitionsExchangeFuture.java:894)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:769)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:715)
>         at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:472)
>         at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1333)
>         ... 2 more
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message