ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergey Sergeev <webs.supp...@gmail.com>
Subject Re: And again... Failed to get page IO instance (page content is corrupted)
Date Fri, 09 Mar 2018 09:31:18 GMT
Hi Mikhail,

Unfortunately, the problem has repeated itself on ignite-core-2.3.3

27.02.18 00:27:55 ERROR  GridCacheIoManager - Failed to process message
[senderId=8f99c887-cd4b-4c38-a649-ca430040d535, messageType=class
o.a.i.i.processors.cache.distributed.dht.atomic.
GridNearAtomicUpdateResponse]
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
IgniteCacheOffheapManagerImpl.expire(IgniteCacheOffheapManagerImpl.java:979)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
*GridCacheTtlManager.expire*(GridCacheTtlManager.java:197)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GridCacheUtils.unwindEvicts(GridCacheUtils.java:833)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
onMessageProcessed(GridCacheIoManager.java:1099)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
processMessage(GridCacheIoManager.java:1072) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GridCacheIoManager.onMessage0(GridCacheIoManager.java:579)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
handleMessage(GridCacheIoManager.java:378) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager.
handleMessage(GridCacheIoManager.java:304) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GridCacheIoManager.access$100(GridCacheIoManager.java:99)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.
onMessage(GridCacheIoManager.java:293) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.
GridIoManager.invokeListener(GridIoManager.java:1555)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.GridIoManager.
processRegularMessage0(GridIoManager.java:1183)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.
GridIoManager.access$4200(GridIoManager.java:126)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.managers.communication.
GridIoManager$9.run(GridIoManager.java:1090) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:505)
~[ignite-core-2.3.3.jar:2.3.3]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: java.lang.IllegalStateException: Failed to get page IO instance
(page content is corrupted)
        at org.apache.ignite.internal.processors.cache.persistence.
tree.io.IOVersions.forVersion(IOVersions.java:83)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.io.IOVersions.forPage(IOVersions.java:95)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.
PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.
PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.
PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
        ... 17 more


And if we are reading from cache...


27.02.18 00:27:56 ERROR MessagePartProcessingHandler - error processing
incoming messagePartDeliveryReceipt=MessagePartDeliveryReceipt{id=null,
abonent=7****, externalId='7586F73B'}
org.springframework.dao.DataAccessResourceFailureException: Runtime failure
on bounds: [lower=null, upper=PendingRow []]; nested exception is class
org.apache.ignite.IgniteException: Runtime failure on bounds: [lower=null,
upper=PendingRow []]
        at com.a1s.sdp.data.dao.ignite.IgniteExceptionTranslator.
translateExceptionIfPossible(IgniteExceptionTranslator.java:35)
~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:41)
~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.
findOne(BaseIgniteCassandraRepository.java:136)
~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.service.TransportServiceImpl.
processMessagePartDelivered(TransportServiceImpl.java:213)
~[core-common-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.service.TransportServiceImpl.
messagePartDelivered(TransportServiceImpl.java:202)
~[core-common-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.common.transport.handler.
MessagePartProcessingHandler.lambda$channelReadMessagePartDeliveryReceipt$1(
MessagePartProcessingHandler.java:91) ~[core-common-1.5-SNAPSHOT.jar:3752]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[?:1.8.0_151]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
~[?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
~[?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:*1.8.0_151*]
Caused by: org.apache.ignite.IgniteException: *Runtime failure on bounds:
[lower=null, upper=PendingRow []]*
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:954) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
*IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GridCacheUtils.unwindEvicts(GridCacheUtils.java:833)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GridCacheGateway.leaveNoLock(GridCacheGateway.java:240)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GridCacheGateway.leave(GridCacheGateway.java:225)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GatewayProtectedCacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:608)
~[ignite-core-2.3.3.jar:2.3.3]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.
lambda$findOne$7(BaseIgniteCassandraRepository.java:136)
~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39)
~[core-data-1.5-SNAPSHOT.jar:3752]
        ... 9 more
Caused by: java.lang.IllegalStateException: Failed to get page IO instance
(page content is corrupted)
        at org.apache.ignite.internal.processors.cache.persistence.
tree.io.IOVersions.forVersion(IOVersions.java:83)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.io.IOVersions.forPage(IOVersions.java:95)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.
PendingRow.initKey(PendingRow.java:72) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.
PendingEntriesTree.getRow(PendingEntriesTree.java:118)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.tree.
PendingEntriesTree.getRow(PendingEntriesTree.java:31)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.fillFromBuffer(BPlusTree.java:4539)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.init(BPlusTree.java:4441)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree$ForwardCursor.access$5300(BPlusTree.java:4380)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.findLowerUnbounded(BPlusTree.java:910)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:942) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.persistence.
tree.BPlusTree.find(BPlusTree.java:933) ~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
*IgniteCacheOffheapManagerImpl.**expire*(IgniteCacheOffheapManagerImpl.java:979)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GridCacheTtlManager.expire(GridCacheTtlManager.java:197)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GridCacheUtils.unwindEvicts(GridCacheUtils.java:833)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GridCacheGateway.leaveNoLock(GridCacheGateway.java:240)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GridCacheGateway.leave(GridCacheGateway.java:225)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GatewayProtectedCacheProxy.onLeave(GatewayProtectedCacheProxy.java:1510)
~[ignite-core-2.3.3.jar:2.3.3]
        at org.apache.ignite.internal.processors.cache.
GatewayProtectedCacheProxy.get(GatewayProtectedCacheProxy.java:608)
~[ignite-core-2.3.3.jar:2.3.3]
        at com.a1s.sdp.data.dao.ignite.BaseIgniteCassandraRepository.
lambda$findOne$7(BaseIgniteCassandraRepository.java:136)
~[core-data-1.5-SNAPSHOT.jar:3752]
        at com.a1s.sdp.data.dao.ignite.IgniteDaoHelper.wrapIgnite(IgniteDaoHelper.java:39)
~[core-data-1.5-SNAPSHOT.jar:3752]
        ... 9 more


The attachment (https://yadi.sk/d/u7fSK8Y53SpCpA) contains a part of cache
obtained from node of cluster - /var/ignite/***/db/
node00-66fd86f5-4e2d-4f1b-8fe1-993bf5937d99/cache-MessagePartSent.
This cache contains objects of class MessagePartSent:

@Table("message_part_sent_external")
public final class MessagePartSent {
    @Id
    private MessagePartSentPk id;
    private MessagePartPk partId;

    public MessagePartSentPk getId() {
        return id;
    }

    public void setId(MessagePartSentPk id) {
        this.id = id;
    }

    public MessagePartPk getPartId() {
        return partId;
    }

    public void setPartId(MessagePartPk partId) {
        this.partId = partId;
    }

    @Override
    public String toString() {
        final StringBuilder sb = new StringBuilder("MessagePartSent{");
        sb.append("id=").append(id);
        sb.append(", partId=").append(partId);
        sb.append('}');
        return sb.toString();
    }

    public static final class MessagePartSentPk implements Serializable {
        private Integer linkId;
        private String externalId;

        public MessagePartSentPk() {
            //
        }

        public MessagePartSentPk(Integer linkId, String externalId) {
            this.linkId = linkId;
            this.externalId = externalId;
        }

        public Integer getLinkId() {
            return linkId;
        }

        public void setLinkId(Integer linkId) {
            this.linkId = linkId;
        }

        public String getExternalId() {
            return externalId;
        }

        public void setExternalId(String externalId) {
            this.externalId = externalId;
        }

        @Override
        public boolean equals(Object o) {
            if (this == o) return true;
            if (!(o instanceof MessagePartSentPk)) return false;

            final MessagePartSentPk that = (MessagePartSentPk) o;

            if (externalId != null ? !externalId.equals(that.externalId) :
that.externalId != null) return false;
            if (linkId != null ? !linkId.equals(that.linkId) : that.linkId
!= null) return false;

            return true;
        }

        @Override
        public int hashCode() {
            int result = linkId != null ? linkId.hashCode() : 0;
            result = 31 * result + (externalId != null ?
externalId.hashCode() : 0);
            return result;
        }

        @Override
        public String toString() {
            final StringBuilder sb = new StringBuilder("
MessagePartSentPk{");
            sb.append("linkId=").append(linkId);
            sb.append(", externalId='").append(externalId).append('\'');
            sb.append('}');
            return sb.toString();
        }
    }

}


public final class MessagePartPk implements Serializable {
    private UUID transactionId;
    private Short partNumber;

    public MessagePartPk() {
        //default
    }

    public MessagePartPk(UUID transactionId, Short partNumber) {
        this.transactionId = transactionId;
        this.partNumber = partNumber;
    }

    public UUID getTransactionId() {
        return transactionId;
    }

    public void setTransactionId(UUID transactionId) {
        this.transactionId = transactionId;
    }

    public Short getPartNumber() {
        return partNumber;
    }

    public void setPartNumber(Short partNumber) {
        this.partNumber = partNumber;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (!(o instanceof MessagePartPk)) return false;

        final MessagePartPk messagePartPk = (MessagePartPk) o;

        if (partNumber != null ? !partNumber.equals(messagePartPk.partNumber)
:
                messagePartPk.partNumber != null) return false;
        if (transactionId != null ?
!transactionId.equals(messagePartPk.transactionId)
:
                messagePartPk.transactionId != null)
            return false;

        return true;
    }

    @Override
    public int hashCode() {
        int result = transactionId != null ? transactionId.hashCode() : 0;
        result = 31 * result + (partNumber != null ? partNumber.hashCode()
: 0);
        return result;
    }

    @Override
    public String toString() {
        final StringBuilder sb = new StringBuilder("MessagePartPk{");
        sb.append("transactionId=").append(transactionId);
        sb.append(", partNumber=").append(partNumber);
        sb.append('}');
        return sb.toString();
}

On Thu, Mar 8, 2018 at 11:03 PM, Dmitriy Setrakyan <dsetrakyan@apache.org>
wrote:

> Hi Sergey,
>
> The 2.4 release is about to be voted for. You can use the RC1 in the mean
> time:
> http://apache-ignite-developers.2346864.n4.nabble.
> com/VOTE-Apache-Ignite-2-4-0-RC1-td27687.html
>
> D.
>
> On Mon, Feb 19, 2018 at 6:43 AM, Mikhail <michael.cherkasov@gmail.com>
> wrote:
>
>> Hi Sergey,
>>
>> The release of 2.4 should be soon, in a week or couple, however, there's
>> no
>> strong schedule for Apache releases.
>>
>> Could you please share a reproducer for the issue? Might be you can share
>> a
>> storage on which the issue can be reproduced?
>>
>> Thanks,
>> Mike.
>>
>>
>>
>> --
>> Sent from: http://apache-ignite-users.70518.x6.nabble.com/
>>
>
>


-- 
Best regards,
Sergey S. Sergeev (aka SSSerj)
Notes: https://twitter.com/ssserj
exit(0);

Mime
View raw message