From issues-return-88997-archive-asf-public=cust-asf.ponee.io@ignite.apache.org Wed Jan 23 18:18:52 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 23B3218077A for ; Wed, 23 Jan 2019 18:18:50 +0100 (CET) Received: (qmail 49175 invoked by uid 500); 23 Jan 2019 17:18:50 -0000 Mailing-List: contact issues-help@ignite.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@ignite.apache.org Delivered-To: mailing list issues@ignite.apache.org Received: (qmail 49165 invoked by uid 99); 23 Jan 2019 17:18:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 23 Jan 2019 17:18:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id D3716C0BFA for ; Wed, 23 Jan 2019 17:18:49 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id R1YgIpfwvriC for ; Wed, 23 Jan 2019 17:18:47 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 286C25FDA3 for ; Wed, 23 Jan 2019 17:12:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 55153E26A5 for ; Wed, 23 Jan 2019 17:12:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 61E61242A8 for ; Wed, 23 Jan 2019 17:12:00 +0000 (UTC) Date: Wed, 23 Jan 2019 17:12:00 +0000 (UTC) From: "Dmitriy Pavlov (JIRA)" To: issues@ignite.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Comment Edited] (IGNITE-11030) Teamcity Bot cache is in unrecoverable state after bot restart: need to check reasons MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/IGNITE-11030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16750235#comment-16750235 ] Dmitriy Pavlov edited comment on IGNITE-11030 at 1/23/19 5:11 PM: ------------------------------------------------------------------ Scanning segment [248190], relIdx [0] InitNewPageRecord:: pos=FileWALPointer [idx=248190, fileOff=7691736, len=45], rec=InitNewPageRecord [newPageId=00010000000145b4, super=PageDeltaRecord [grpId=482599155, pageId=0101001a000145b4, super=WALRecord [size=45, chainSize=0, pos=FileWALPointer [idx=248190, fileOff=7691736, len=45], type=INIT_NEW_PAGE_RECORD]]], ioType=1, ioVersion=1[2019-01-23 19:31:04,287][ERROR][main][root] Test failed. has different partitions: pageId=101001a000145b4 newPageId= 10000000145b4 was (Author: dpavlov): pageId=101001a000145b4 newPageId= 10000000145b4 > Teamcity Bot cache is in unrecoverable state after bot restart: need to check reasons > ------------------------------------------------------------------------------------- > > Key: IGNITE-11030 > URL: https://issues.apache.org/jira/browse/IGNITE-11030 > Project: Ignite > Issue Type: Task > Reporter: Dmitriy Pavlov > Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > - TC Bot was stopped with kill command (default signal). > - Shutdown hook calls ignite close > But after updating to V190122 TC Bot DB was became unrecoverable for "teamcityTestRunHist" cache with exception > {noformat} > Failure periodic check failed: javax.cache.CacheException: class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on lookup row: SearchRow [key=org.apache.ignite.ci.teamcity.ignited.runhist.RunHistKey [idHash=1028871081, hash=1241170874, srvId=1411517106, testOrSuiteName=11924, branch=281], hash=1241170874, cacheId=0] > com.google.common.util.concurrent.UncheckedExecutionException: javax.cache.CacheException: class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on lookup row: SearchRow [key=org.apache.ignite.ci.teamcity.ignited.runhist.RunHistKey [idHash=1028871081, hash=1241170874, srvId=1411517106, testOrSuiteName=11924, branch=281], hash=1241170874, cacheId=0] > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2050) > at com.google.common.cache.LocalCache.get(LocalCache.java:3952) > at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4871) > at org.apache.ignite.ci.di.cache.GuavaCachedInterceptor.invoke(GuavaCachedInterceptor.java:59) > at org.apache.ignite.ci.teamcity.ignited.TeamcityIgnitedImpl.getTestRunHist(TeamcityIgnitedImpl.java:404) > at org.apache.ignite.ci.di.AutoProfilingInterceptor.invoke(AutoProfilingInterceptor.java:76) > at org.apache.ignite.ci.web.model.current.SuiteCurrentStatus.lambda$initFromContext$0(SuiteCurrentStatus.java:155) > at java.util.Comparator.lambda$comparing$77a9974f$1(Comparator.java:469) > at java.util.Collections$ReverseComparator2.compare(Collections.java:5178) > at java.util.TimSort.countRunAndMakeAscending(TimSort.java:355) > at java.util.TimSort.sort(TimSort.java:220) > at java.util.Arrays.sort(Arrays.java:1512) > at java.util.ArrayList.sort(ArrayList.java:1462) > at org.apache.ignite.ci.web.model.current.SuiteCurrentStatus.initFromContext(SuiteCurrentStatus.java:160) > at org.apache.ignite.ci.web.model.current.ChainAtServerCurrentStatus.lambda$initFromContext$0(ChainAtServerCurrentStatus.java:171) > at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184) > at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) > at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) > at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) > at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) > at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) > at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) > at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) > at org.apache.ignite.ci.web.model.current.ChainAtServerCurrentStatus.initFromContext(ChainAtServerCurrentStatus.java:167) > at org.apache.ignite.ci.tcbot.chain.TrackedBranchChainsProcessor.lambda$getTrackedBranchTestFailures$1(TrackedBranchChainsProcessor.java:121) > at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) > at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) > at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382) > at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481) > at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) > at java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151) > at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174) > at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) > at org.apache.ignite.ci.tcbot.chain.TrackedBranchChainsProcessor.getTrackedBranchTestFailures(TrackedBranchChainsProcessor.java:125) > at org.apache.ignite.ci.di.AutoProfilingInterceptor.invoke(AutoProfilingInterceptor.java:76) > at org.apache.ignite.ci.tcbot.issue.IssueDetector.checkFailuresEx(IssueDetector.java:445) > at org.apache.ignite.ci.di.MonitoredTaskInterceptor.invoke(MonitoredTaskInterceptor.java:111) > at org.apache.ignite.ci.di.AutoProfilingInterceptor.invoke(AutoProfilingInterceptor.java:76) > at org.apache.ignite.ci.tcbot.issue.IssueDetector.checkFailures(IssueDetector.java:422) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: javax.cache.CacheException: class org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on lookup row: SearchRow [key=org.apache.ignite.ci.teamcity.ignited.runhist.RunHistKey [idHash=1028871081, hash=1241170874, srvId=1411517106, testOrSuiteName=11924, branch=281], hash=1241170874, cacheId=0] > at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1337) > at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.cacheException(IgniteCacheProxyImpl.java:1758) > at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:931) > at org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:640) > at org.apache.ignite.ci.teamcity.ignited.runhist.RunHistCompactedDao.getTestRunHist(RunHistCompactedDao.java:102) > at org.apache.ignite.ci.di.cache.GuavaCachedInterceptor$1.call(GuavaCachedInterceptor.java:64) > at org.apache.ignite.ci.di.cache.GuavaCachedInterceptor$1.call(GuavaCachedInterceptor.java:60) > at com.google.common.cache.LocalCache$LocalManualCache$1.load(LocalCache.java:4876) > at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3528) > at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2277) > at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2154) > at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2044) > ... 47 common frames omitted > Caused by: org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: Runtime failure on lookup row: SearchRow [key=org.apache.ignite.ci.teamcity.ignited.runhist.RunHistKey [idHash=1028871081, hash=1241170874, srvId=1411517106, testOrSuiteName=11924, branch=281], hash=1241170874, cacheId=0] > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne(BPlusTree.java:1251) > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne(BPlusTree.java:1227) > at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.find(IgniteCacheOffheapManagerImpl.java:2754) > at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.find(GridCacheOffheapManager.java:1953) > at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.read(IgniteCacheOffheapManagerImpl.java:670) > at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.localGet(GridPartitionedSingleGetFuture.java:395) > at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.mapKeyToNode(GridPartitionedSingleGetFuture.java:353) > at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.map(GridPartitionedSingleGetFuture.java:224) > at org.apache.ignite.internal.processors.cache.distributed.dht.GridPartitionedSingleGetFuture.init(GridPartitionedSingleGetFuture.java:216) > at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.getAsync0(GridDhtAtomicCache.java:1428) > at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$1600(GridDhtAtomicCache.java:135) > at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$16.apply(GridDhtAtomicCache.java:474) > at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$16.apply(GridDhtAtomicCache.java:472) > at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.asyncOp(GridDhtAtomicCache.java:761) > at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.getAsync(GridDhtAtomicCache.java:472) > at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get0(GridCacheAdapter.java:4716) > at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:4697) > at org.apache.ignite.internal.processors.cache.GridCacheAdapter.get(GridCacheAdapter.java:1415) > at org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.get(IgniteCacheProxyImpl.java:928) > ... 56 common frames omitted > Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted) > at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:85) > at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:97) > at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:154) > at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:108) > at org.apache.ignite.internal.processors.cache.tree.DataRow.(DataRow.java:55) > at org.apache.ignite.internal.processors.cache.tree.CacheDataRowStore.dataRow(CacheDataRowStore.java:92) > at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:200) > at org.apache.ignite.internal.processors.cache.tree.CacheDataTree.getRow(CacheDataTree.java:49) > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetOne.found(BPlusTree.java:2862) > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:298) > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5615) > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run(BPlusTree.java:271) > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:5600) > at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:159) > at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:334) > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1309) > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1318) > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findDown(BPlusTree.java:1318) > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doFind(BPlusTree.java:1276) > at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findOne(BPlusTree.java:1243) > ... 74 common frames omitted > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)