ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anton Kalashnikov (Jira)" <j...@apache.org>
Subject [jira] [Commented] (IGNITE-11939) IgnitePdsTxHistoricalRebalancingTest.testTopologyChangesWithConstantLoad test failure
Date Tue, 26 Nov 2019 13:31:00 GMT

    [ https://issues.apache.org/jira/browse/IGNITE-11939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16982473#comment-16982473
] 

Anton Kalashnikov commented on IGNITE-11939:
--------------------------------------------

I figured out the following things:
First of all, we are trying to release a segment from two places(exchange and full message
handler):
{noformat}
org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentReservationStorage.release(SegmentReservationStorage.java:101)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentAware.release(SegmentAware.java:219)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.release(FileWriteAheadLogManager.java:997)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.releaseHistoryForPreloading(GridCacheDatabaseSharedManager.java
:1865)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.processFullPartitionUpdate(GridCachePartitionExchangeManager.java:1829)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$3.onMessage(GridCachePartitionExchangeManager.java:441)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$3.onMessage(GridCachePartitionExchangeManager.java:428)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3622)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$MessageHandler.apply(GridCachePartitionExchangeManager.java:3601)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:1142)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:591)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:392)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:318)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$100(GridCacheIoManager.java:109)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:308)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1843)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1468)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.managers.communication.GridIoManager.access$5200(GridIoManager.java:229)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1365)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] java.lang.Thread.run(Thread.java:748)
{noformat}
{noformat}
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentReservationStorage.release(SegmentReservationStorage.java:101)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentAware.release(SegmentAware.java:219)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.release(FileWriteAheadLogManager.java:997)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.releaseHistoryForPreloading(GridCacheDatabaseSharedManager.java:1865)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1438)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:882)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3172)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:3021)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
[16:35:58]W:             [org.apache.ignite:ignite-indexing] java.lang.Thread.run(Thread.java:748)
{noformat}

But as I can see method GridCacheDatabaseSharedManager#releaseHistoryForPreloading doesn't
design for concurrency usage due to the count of iteration through reservedForPreloading will
be unpredictable. So if add synchronization to this method it'll fix that problem(https://ci.ignite.apache.org/buildConfiguration/IgniteTests24Java8_PdsIndexing?branch=pull%2F7066%2Fhead&buildTypeTab=overview)
Take a look at releaseHistoryLock bellow:
{noformat}
private final ReentrantLock releaseHistoryLock = new ReentrantLock();

    /** {@inheritDoc} */
    @Override public void releaseHistoryForPreloading() {
        releaseHistoryLock.lock();
        try {
            for (Map.Entry<T2<Integer, Integer>, T2<Long, WALPointer>> e
: reservedForPreloading.entrySet()) {
                try {
                    cctx.wal().release(e.getValue().get2());
                }
                catch (IgniteCheckedException ex) {
                    U.error(log, "Could not release WAL reservation", ex);

                    throw new IgniteException(ex);
                }
            }

            reservedForPreloading.clear();
        }
        finally {
            releaseHistoryLock.unlock();
        }
    }
{noformat}

[~ivan.glukos] Can you help with the next steps to solve this problem?

>  IgnitePdsTxHistoricalRebalancingTest.testTopologyChangesWithConstantLoad test failure
> --------------------------------------------------------------------------------------
>
>                 Key: IGNITE-11939
>                 URL: https://issues.apache.org/jira/browse/IGNITE-11939
>             Project: Ignite
>          Issue Type: Bug
>            Reporter: Alexei Scherbakov
>            Assignee: Anton Kalashnikov
>            Priority: Major
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> Caused by exception on releasing reserved segments:
> {noformat}
> [12:51:23]W:             [org.apache.ignite:ignite-indexing] [2019-06-21 12:51:23,967][ERROR][exchange-worker-#33825%persistence.IgnitePdsTxHistoricalRebalancingTest1%][GridDhtPartitionsExchangeFuture]
Failed to reinitialize local partitions (rebalancing will be stopped)
> : GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=7, minorTopVer=1],
discoEvt=DiscoveryCustomEvent [customMsg=CacheAffinityChangeMessage [id=08de0ff7b61-276ac575-e4dc-4525-b24b-d0a5d1d7633d,
topVer=AffinityTopologyVersion [topVer=7, minorTopVer=0], exc
> hId=null, partsMsg=null, exchangeNeeded=true], affTopVer=AffinityTopologyVersion [topVer=7,
minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=97e46568-6aa0-4a4b-864c-f05415c00000,
consistentId=persistence.IgnitePdsTxHistoricalRebalancingTest0, addrs=Arra
> yList [127.0.0.1], sockAddrs=HashSet [/127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1561110643882, loc=false, ver=2.8.0#20190621-sha1:00000000, isClient=false],
topVer=7, nodeId8=0ff3354e, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=15611106839
> 58]], nodeId=97e46568, evt=DISCOVERY_CUSTOM_EVT]
> [12:51:23]W:             [org.apache.ignite:ignite-indexing] java.lang.AssertionError:
cur=null, absIdx=0
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentReservationStorage.release(SegmentReservationStorage.java:55)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentAware.release(SegmentAware.java:207)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.release(FileWriteAheadLogManager.java:983)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.releaseHistoryForPreloading(GridCacheDatabaseSharedManager.java:1844)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1431)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:862)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3079)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2928)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at java.lang.Thread.run(Thread.java:748)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing] [12:51:23] (err) Failed
to notify listener: o.a.i.i.processors.timeout.GridTimeoutProcessor$2@79ba1907java.lang.AssertionError:
cur=null, absIdx=0
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentReservationStorage.release(SegmentReservationStorage.java:55)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.persistence.wal.aware.SegmentAware.release(SegmentAware.java:207)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.release(FileWriteAheadLogManager.java:983)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.releaseHistoryForPreloading(GridCacheDatabaseSharedManager.java:1844)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1431)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:862)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body0(GridCachePartitionExchangeManager.java:3079)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2928)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> [12:51:23]W:             [org.apache.ignite:ignite-indexing]    at java.lang.Thread.run(Thread.java:748)
> {noformat}
> Timed out run link [1]
> [1] https://ci.ignite.apache.org/viewLog.html?buildId=4168883&buildTypeId=IgniteTests24Java8_PdsIndexing



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message