ignite-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ozerov (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (IGNITE-11217) After cluster kill and restart with new jvm option - one of node fail with failure type - Failed to read WAL record at position
Date Mon, 11 Mar 2019 07:07:00 GMT

     [ https://issues.apache.org/jira/browse/IGNITE-11217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Vladimir Ozerov updated IGNITE-11217:
-------------------------------------
    Labels: mvcc_stability  (was: )

> After cluster kill and restart with new jvm option - one of node fail with failure type
- Failed to read WAL record at position 
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: IGNITE-11217
>                 URL: https://issues.apache.org/jira/browse/IGNITE-11217
>             Project: Ignite
>          Issue Type: Bug
>          Components: data structures, mvcc
>    Affects Versions: 2.7
>            Reporter: ARomantsov
>            Priority: Critical
>              Labels: mvcc_stability
>             Fix For: 2.8
>
>
> Test scenario:
> - Start two server load
> - execute operations (put/put-all/streamer/remove)
> - execute client that collect checksum
> - kill nodes
> - add JVM options to server - DIGNITE_WAL_LOG_TX_RECORDS=true, DIGNITE_RECOVERY_VERBOSE_LOGGING
> - start nodes
> - activate them
> Found that one node fail
> {code:java}
> [23:16:06,230][INFO][main][GridCacheProcessor] Finished restoring partition state for
local groups [groupsProcessed=45, partitionsProcessed=764, time=557ms]
> [23:16:06,280][INFO][main][FileWriteAheadLogManager] Stopping WAL iteration due to an
exception: Failed to read WAL record at position: 27700852 size: 67108864, ptr=FileWALPointer
[idx=0, fileOff=27700852, len=0]
> [23:16:06,298][SEVERE][main][IgniteKernal] Exception during start processors, node will
be stopped and close connections
> class org.apache.ignite.IgniteCheckedException: Failed to read WAL record at position:
27700852 size: 67108864
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.handleRecordException(AbstractWalRecordsIterator.java:305)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.handleRecordException(FileWriteAheadLogManager.java:2691)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advanceRecord(AbstractWalRecordsIterator.java:269)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advance(AbstractWalRecordsIterator.java:158)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.onNext(AbstractWalRecordsIterator.java:127)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.onNext(AbstractWalRecordsIterator.java:53)
> 	at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.nextX(GridCloseableIteratorAdapter.java:41)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.tailPointer(GridCacheDatabaseSharedManager.java:1984)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:1958)
> 	at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1050)
> 	at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040)
> 	at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732)
> 	at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> 	at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
> 	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
> 	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
> 	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
> 	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
> 	at org.apache.ignite.Ignition.start(Ignition.java:348)
> 	at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read WAL record
at position: 27700852 size: 67108864
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:395)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV2Serializer.readRecord(RecordV2Serializer.java:236)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advanceRecord(AbstractWalRecordsIterator.java:254)
> 	... 17 more
> Caused by: java.io.IOException: class org.apache.ignite.IgniteCheckedException: Unsupported
protocol version: 34
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readVersion(RecordDataV1Serializer.java:2010)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV2Serializer.readMvccDataEntry(RecordDataV2Serializer.java:328)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV2Serializer.readPlainRecord(RecordDataV2Serializer.java:162)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readRecord(RecordDataV1Serializer.java:208)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV2Serializer$2.readWithHeaders(RecordV2Serializer.java:163)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:374)
> 	... 19 more
> 	Suppressed: class org.apache.ignite.internal.processors.cache.persistence.wal.crc.IgniteDataIntegrityViolationException:
val: 1816937838 writtenCrc: 1387698722
> 		at org.apache.ignite.internal.processors.cache.persistence.wal.io.FileInput$Crc32CheckingFileInput.close(FileInput.java:106)
> 		at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:381)
> 		... 19 more
> Caused by: class org.apache.ignite.IgniteCheckedException: Unsupported protocol version:
34
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.CacheVersionIO.checkProtocolVersion(CacheVersionIO.java:143)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.CacheVersionIO.readSize(CacheVersionIO.java:156)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readVersion(RecordDataV1Serializer.java:2003)
> 	... 24 more
> [23:16:06,299][SEVERE][main][IgniteKernal] Got exception while starting (will rollback
startup routine).
> class org.apache.ignite.IgniteCheckedException: Failed to read WAL record at position:
27700852 size: 67108864
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.handleRecordException(AbstractWalRecordsIterator.java:305)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.handleRecordException(FileWriteAheadLogManager.java:2691)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advanceRecord(AbstractWalRecordsIterator.java:269)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advance(AbstractWalRecordsIterator.java:158)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.onNext(AbstractWalRecordsIterator.java:127)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.onNext(AbstractWalRecordsIterator.java:53)
> 	at org.apache.ignite.internal.util.GridCloseableIteratorAdapter.nextX(GridCloseableIteratorAdapter.java:41)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.tailPointer(GridCacheDatabaseSharedManager.java:1984)
> 	at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.startMemoryRestore(GridCacheDatabaseSharedManager.java:1958)
> 	at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1050)
> 	at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2040)
> 	at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1732)
> 	at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1158)
> 	at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1076)
> 	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:962)
> 	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:861)
> 	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:731)
> 	at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:700)
> 	at org.apache.ignite.Ignition.start(Ignition.java:348)
> 	at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301)
> Caused by: class org.apache.ignite.IgniteCheckedException: Failed to read WAL record
at position: 27700852 size: 67108864
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:395)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV2Serializer.readRecord(RecordV2Serializer.java:236)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.AbstractWalRecordsIterator.advanceRecord(AbstractWalRecordsIterator.java:254)
> 	... 17 more
> Caused by: java.io.IOException: class org.apache.ignite.IgniteCheckedException: Unsupported
protocol version: 34
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readVersion(RecordDataV1Serializer.java:2010)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV2Serializer.readMvccDataEntry(RecordDataV2Serializer.java:328)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV2Serializer.readPlainRecord(RecordDataV2Serializer.java:162)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readRecord(RecordDataV1Serializer.java:208)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV2Serializer$2.readWithHeaders(RecordV2Serializer.java:163)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:374)
> 	... 19 more
> 	Suppressed: class org.apache.ignite.internal.processors.cache.persistence.wal.crc.IgniteDataIntegrityViolationException:
val: 1816937838 writtenCrc: 1387698722
> 		at org.apache.ignite.internal.processors.cache.persistence.wal.io.FileInput$Crc32CheckingFileInput.close(FileInput.java:106)
> 		at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordV1Serializer.readWithCrc(RecordV1Serializer.java:381)
> 		... 19 more
> Caused by: class org.apache.ignite.IgniteCheckedException: Unsupported protocol version:
34
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.CacheVersionIO.checkProtocolVersion(CacheVersionIO.java:143)
> 	at org.apache.ignite.internal.processors.cache.persistence.tree.io.CacheVersionIO.readSize(CacheVersionIO.java:156)
> 	at org.apache.ignite.internal.processors.cache.persistence.wal.serializer.RecordDataV1Serializer.readVersion(RecordDataV1Serializer.java:2003)
> 	... 24 more
> [23:16:06,302][WARNING][main][IgniteKernal] Attempt to stop starting grid. This operation
cannot be guaranteed to be successful.
> [23:16:06,313][INFO][main][GridTcpRestProtocol] Command protocol successfully stopped:
TCP binary
> [23:16:06,347][SEVERE][ttl-cleanup-worker-#45][] Critical system error detected. Will
be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false,
timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=[SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]],
failureCtx=FailureContext [type=SYSTEM_WORKER_TERMINATION, err=class o.a.i.IgniteException:
Got interrupted while waiting for future to complete.]]
> class org.apache.ignite.IgniteException: Got interrupted while waiting for future to
complete.
> 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.localJoin(GridDiscoveryManager.java:2208)
> 	at org.apache.ignite.internal.processors.cache.GridCacheSharedTtlCleanupManager$CleanupWorker.body(GridCacheSharedTtlCleanupManager.java:136)
> 	at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120)
> 	at java.lang.Thread.run(Thread.java:748)
> Caused by: class org.apache.ignite.internal.IgniteInterruptedCheckedException: Got interrupted
while waiting for future to complete.
> 	at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:186)
> 	at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:141)
> 	at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.localJoin(GridDiscoveryManager.java:2205)
> 	... 3 more
> [23:16:06,347][WARNING][ttl-cleanup-worker-#45][FailureProcessor] No deadlocked threads
detected.
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message