ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexei Scherbakov <alexey.scherbak...@gmail.com>
Subject Re: "Idle verify" to "Online verify"
Date Tue, 07 May 2019 15:10:32 GMT
Anton,

It's ready for review, look for Patch Available status.
Yes, atomic caches are not fixed by this contribution. See [1]

[1] https://issues.apache.org/jira/browse/IGNITE-11797

вт, 7 мая 2019 г. в 17:30, Anton Vinogradov <av@apache.org>:

> Alexei,
>
> Got it.
> Could you please let me know once PR will be ready for review?
> Currently, have some questions, but, possible, they caused by non-final PR
> (eg. why atomic counter still ignores misses?).
>
> On Tue, May 7, 2019 at 4:43 PM Alexei Scherbakov <
> alexey.scherbakoff@gmail.com> wrote:
>
> > Anton,
> >
> > 1) Extended counters indeed will answer the question if partition could
> be
> > safely restored to synchronized state on all owners.
> > The only condition - one of owners has no missed updates.
> > If not, partition must be moved to LOST state, see [1],
> > TxPartitionCounterStateOnePrimaryTwoBackupsFailAll*Test,
> >
> >
> IgniteSystemProperties#IGNITE_FAIL_NODE_ON_UNRECOVERABLE_PARTITION_INCONSISTENCY
> > This is known issue and could happen if all partition owners were
> > unavailable at some point.
> > In such case we could try to recover consistency using some complex
> > recovery protocol as you described. Related ticket [2]
> >
> > 2) Bitset implementation is considered as an option in GG Community
> > Edition. No specific implementation dates at the moment.
> >
> > 3) As for "online" partition verification,  I think the best option right
> > now is to do verification partition by partition using read only mode per
> > group partition under load.
> > While verification is in progress, all write ops are waiting, not
> rejected.
> > This is only 100% reliable way to compare partitions - by touching actual
> > data, all other ways like pre-computed hash are error prone.
> > There is already ticket [3] for simplifing grid consistency verification
> > which could be used as basis for such functionality.
> > As for avoiding cache pollution, we could try read pages sequentially
> from
> > disk without lifting them to pagemem and computing some kind of
> commutative
> > hash. It's safe under partition write lock.
> >
> > [1] https://issues.apache.org/jira/browse/IGNITE-11611
> > [2] https://issues.apache.org/jira/browse/IGNITE-6324
> > [3] https://issues.apache.org/jira/browse/IGNITE-11256
> >
> > пн, 6 мая 2019 г. в 16:12, Anton Vinogradov <av@apache.org>:
> >
> > > Ivan,
> > >
> > > 1) I've checked the PR [1] and it looks like it does not solve the
> issue
> > > too.
> > > AFAICS, the main goal here (at PR) is to produce
> > > PartitionUpdateCounter#sequential which can be false for all backups,
> > what
> > > backup should win in that case?
> > >
> > > Is there any IEP or some another design page for this fix?
> > >
> > > Looks like extended counters should be able to recover the whole
> cluster
> > > even in case all copies of the same partition are broken.
> > > So, seems, the counter should provide detailed info:
> > > - biggest applied updateCounter
> > > - list of all missed counters before biggest applied
> > > - optional hash
> > >
> > > In that case, we'll be able to perform some exchange between broken
> > copies.
> > > For example, we'll found that copy1 missed key1, and copy2 missed key2.
> > > It's pretty simple to fix both copies in that case.
> > > In case all misses can be solved this way, we'll continue cluster
> > > activation like it was not broken before.
> > >
> > > 2) Seems I see the simpler solution to handle misses (than at PR).
> > > Once you have newUpdateCounter > curUpdateCounter + 1, you should add
> > byte
> > > (or int or long (smaplest possible)) value to special structure.
> > > This value will represent delta between newUpdateCounter and
> > > curUpdateCounter in bitmask way.
> > > In case you'll handle updateCounter less that curUpdateCounter, you
> > should
> > > update the value at structure responsible to this delta.
> > > For example, when you have delta "2 to 6", you will have 00000000
> > initially
> > > and 00011111 finally.
> > > Each delta update should be finished with check it completed (value ==
> 31
> > > in this case). Once it finished, it should be removed from the
> structure.
> > > Deltas can and should be reused to solve GC issue.
> > >
> > > What do you think about the proposed solution?
> > >
> > > 3) Hash computation can be an additional extension for extended
> counters,
> > > just one more dimension to be extremely sure everything is ok.
> > > Any objections?
> > >
> > > [1] https://github.com/apache/ignite/pull/5765
> > >
> > > On Mon, May 6, 2019 at 12:48 PM Ivan Rakov <ivan.glukos@gmail.com>
> > wrote:
> > >
> > > > Anton,
> > > >
> > > > Automatic quorum-based partition drop may work as a partial
> workaround
> > > > for IGNITE-10078, but discussed approach surely doesn't replace
> > > > IGNITE-10078 activity. We still don't know what do to when quorum
> can't
> > > > be reached (2 partitions have hash X, 2 have hash Y) and keeping
> > > > extended update counters is the only way to resolve such case.
> > > > On the other hand, precalculated partition hashes validation on PME
> can
> > > > be a good addition to IGNITE-10078 logic: we'll be able to detect
> > > > situations when extended update counters are equal, but for some
> reason
> > > > (bug or whatsoever) partition contents are different.
> > > >
> > > > Best Regards,
> > > > Ivan Rakov
> > > >
> > > > On 06.05.2019 12:27, Anton Vinogradov wrote:
> > > > > Ivan, just to make sure ...
> > > > > The discussed case will fully solve the issue [1] in case we'll
> also
> > > add
> > > > > some strategy to reject partitions with missed updates
> > (updateCnt==Ok,
> > > > > Hash!=Ok).
> > > > > For example, we may use the Quorum strategy, when the majority
> wins.
> > > > > Sounds correct?
> > > > >
> > > > > [1] https://issues.apache.org/jira/browse/IGNITE-10078
> > > > >
> > > > > On Tue, Apr 30, 2019 at 3:14 PM Anton Vinogradov <av@apache.org>
> > > wrote:
> > > > >
> > > > >> Ivan,
> > > > >>
> > > > >> Thanks for the detailed explanation.
> > > > >> I'll try to implement the PoC to check the idea.
> > > > >>
> > > > >> On Mon, Apr 29, 2019 at 8:22 PM Ivan Rakov <ivan.glukos@gmail.com
> >
> > > > wrote:
> > > > >>
> > > > >>>> But how to keep this hash?
> > > > >>> I think, we can just adopt way of storing partition update
> > counters.
> > > > >>> Update counters are:
> > > > >>> 1) Kept and updated in heap, see
> > > > >>> IgniteCacheOffheapManagerImpl.CacheDataStoreImpl#pCntr (accessed
> > > during
> > > > >>> regular cache operations, no page replacement latency issues)
> > > > >>> 2) Synchronized with page memory (and with disk) on every
> > checkpoint,
> > > > >>> see GridCacheOffheapManager#saveStoreMetadata
> > > > >>> 3) Stored in partition meta page, see
> > > > PagePartitionMetaIO#setUpdateCounter
> > > > >>> 4) On node restart, we init onheap counter with value from
disk
> > (for
> > > > the
> > > > >>> moment of last checkpoint) and update it to latest value
during
> WAL
> > > > >>> logical records replay
> > > > >>>
> > > > >>>> 2) PME is a rare operation on production cluster, but,
seems, we
> > > have
> > > > >>>> to check consistency in a regular way.
> > > > >>>> Since we have to finish all operations before the check,
should
> we
> > > > >>>> have fake PME for maintenance check in this case?
> > > > >>>   From my experience, PME happens on prod clusters from time
to
> > time
> > > > >>> (several times per week), which can be enough. In case it's
> needed
> > to
> > > > >>> check consistency more often than regular PMEs occur, we
can
> > > implement
> > > > >>> command that will trigger fake PME for consistency checking.
> > > > >>>
> > > > >>> Best Regards,
> > > > >>> Ivan Rakov
> > > > >>>
> > > > >>> On 29.04.2019 18:53, Anton Vinogradov wrote:
> > > > >>>> Ivan, thanks for the analysis!
> > > > >>>>
> > > > >>>>>> With having pre-calculated partition hash value,
we can
> > > > >>>> automatically detect inconsistent partitions on every
PME.
> > > > >>>> Great idea, seems this covers all broken synс cases.
> > > > >>>>
> > > > >>>> It will check alive nodes in case the primary failed
immediately
> > > > >>>> and will check rejoining node once it finished a rebalance
(PME
> on
> > > > >>>> becoming an owner).
> > > > >>>> Recovered cluster will be checked on activation PME (or
even
> > before
> > > > >>>> that?).
> > > > >>>> Also, warmed cluster will be still warmed after check.
> > > > >>>>
> > > > >>>> Have I missed some cases leads to broken sync except
bugs?
> > > > >>>>
> > > > >>>> 1) But how to keep this hash?
> > > > >>>> - It should be automatically persisted on each checkpoint
(it
> > should
> > > > >>>> not require recalculation on restore, snapshots should
be
> covered
> > > too)
> > > > >>>> (and covered by WAL?).
> > > > >>>> - It should be always available at RAM for every partition
(even
> > for
> > > > >>>> cold partitions never updated/readed on this node) to
be
> > immediately
> > > > >>>> used once all operations done on PME.
> > > > >>>>
> > > > >>>> Can we have special pages to keep such hashes and never
allow
> > their
> > > > >>>> eviction?
> > > > >>>>
> > > > >>>> 2) PME is a rare operation on production cluster, but,
seems, we
> > > have
> > > > >>>> to check consistency in a regular way.
> > > > >>>> Since we have to finish all operations before the check,
should
> we
> > > > >>>> have fake PME for maintenance check in this case?
> > > > >>>>
> > > > >>>> On Mon, Apr 29, 2019 at 4:59 PM Ivan Rakov <
> ivan.glukos@gmail.com
> > > > >>>> <mailto:ivan.glukos@gmail.com>> wrote:
> > > > >>>>
> > > > >>>>      Hi Anton,
> > > > >>>>
> > > > >>>>      Thanks for sharing your ideas.
> > > > >>>>      I think your approach should work in general. I'll
just
> share
> > > my
> > > > >>>>      concerns about possible issues that may come up.
> > > > >>>>
> > > > >>>>      1) Equality of update counters doesn't imply equality
of
> > > > >>>>      partitions content under load.
> > > > >>>>      For every update, primary node generates update
counter and
> > > then
> > > > >>>>      update is delivered to backup node and gets applied
with
> the
> > > > >>>>      corresponding update counter. For example, there
are two
> > > > >>>>      transactions (A and B) that update partition X by
the
> > following
> > > > >>>>      scenario:
> > > > >>>>      - A updates key1 in partition X on primary node
and
> > increments
> > > > >>>>      counter to 10
> > > > >>>>      - B updates key2 in partition X on primary node
and
> > increments
> > > > >>>>      counter to 11
> > > > >>>>      - While A is still updating another keys, B is finally
> > > committed
> > > > >>>>      - Update of key2 arrives to backup node and sets
update
> > counter
> > > > to
> > > > >>> 11
> > > > >>>>      Observer will see equal update counters (11), but
update of
> > > key 1
> > > > >>>>      is still missing in the backup partition.
> > > > >>>>      This is a fundamental problem which is being solved
here:
> > > > >>>>      https://issues.apache.org/jira/browse/IGNITE-10078
> > > > >>>>      "Online verify" should operate with new complex
update
> > counters
> > > > >>>>      which take such "update holes" into account. Otherwise,
> > online
> > > > >>>>      verify may provide false-positive inconsistency
reports.
> > > > >>>>
> > > > >>>>      2) Acquisition and comparison of update counters
is fast,
> but
> > > > >>>>      partition hash calculation is long. We should check
that
> > update
> > > > >>>>      counter remains unchanged after every K keys handled.
> > > > >>>>
> > > > >>>>      3)
> > > > >>>>
> > > > >>>>>      Another hope is that we'll be able to pause/continue
scan,
> > for
> > > > >>>>>      example, we'll check 1/3 partitions today, 1/3
tomorrow,
> and
> > > in
> > > > >>>>>      three days we'll check the whole cluster.
> > > > >>>>      Totally makes sense.
> > > > >>>>      We may find ourselves into a situation where some
"hot"
> > > > partitions
> > > > >>>>      are still unprocessed, and every next attempt to
calculate
> > > > >>>>      partition hash fails due to another concurrent update.
We
> > > should
> > > > >>>>      be able to track progress of validation (% of calculation
> > time
> > > > >>>>      wasted due to concurrent operations may be a good
metric,
> > 100%
> > > is
> > > > >>>>      the worst case) and provide option to stop/pause
activity.
> > > > >>>>      I think, pause should return an "intermediate results
> report"
> > > > with
> > > > >>>>      information about which partitions have been successfully
> > > > checked.
> > > > >>>>      With such report, we can resume activity later:
partitions
> > from
> > > > >>>>      report will be just skipped.
> > > > >>>>
> > > > >>>>      4)
> > > > >>>>
> > > > >>>>>      Since "Idle verify" uses regular pagmem, I assume
it
> > replaces
> > > > hot
> > > > >>>>>      data with persisted.
> > > > >>>>>      So, we have to warm up the cluster after each
check.
> > > > >>>>>      Are there any chances to check without cooling
the
> cluster?
> > > > >>>>      I don't see an easy way to achieve it with our page
memory
> > > > >>>>      architecture. We definitely can't just read pages
from disk
> > > > >>>>      directly: we need to synchronize page access with
> concurrent
> > > > >>>>      update operations and checkpoints.
> > > > >>>>      From my point of view, the correct way to solve
this issue
> is
> > > > >>>>      improving our page replacement [1] mechanics by
making it
> > truly
> > > > >>>>      scan-resistant.
> > > > >>>>
> > > > >>>>      P. S. There's another possible way of achieving
online
> > verify:
> > > > >>>>      instead of on-demand hash calculation, we can always
keep
> > > > >>>>      up-to-date hash value for every partition. We'll
need to
> > update
> > > > >>>>      hash on every insert/update/remove operation, but
there
> will
> > be
> > > > no
> > > > >>>>      reordering issues as per function that we use for
> aggregating
> > > > hash
> > > > >>>>      results (+) is commutative. With having pre-calculated
> > > partition
> > > > >>>>      hash value, we can automatically detect inconsistent
> > partitions
> > > > on
> > > > >>>>      every PME. What do you think?
> > > > >>>>
> > > > >>>>      [1] -
> > > > >>>>
> > > > >>>
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/IGNITE/Ignite+Durable+Memory+-+under+the+hood#IgniteDurableMemory-underthehood-Pagereplacement(rotationwithdisk)
> > > > >>>>      Best Regards,
> > > > >>>>      Ivan Rakov
> > > > >>>>
> > > > >>>>      On 29.04.2019 12:20, Anton Vinogradov wrote:
> > > > >>>>>      Igniters and especially Ivan Rakov,
> > > > >>>>>
> > > > >>>>>      "Idle verify" [1] is a really cool tool, to
make sure that
> > > > >>>>>      cluster is consistent.
> > > > >>>>>
> > > > >>>>>      1) But it required to have operations paused
during
> cluster
> > > > check.
> > > > >>>>>      At some clusters, this check requires hours
(3-4 hours at
> > > cases
> > > > I
> > > > >>>>>      saw).
> > > > >>>>>      I've checked the code of "idle verify" and it
seems it
> > > possible
> > > > >>>>>      to make it "online" with some assumptions.
> > > > >>>>>
> > > > >>>>>      Idea:
> > > > >>>>>      Currently "Idle verify" checks that partitions
hashes,
> > > generated
> > > > >>>>>      this way
> > > > >>>>>      while (it.hasNextX()) {
> > > > >>>>>      CacheDataRow row = it.nextX();
> > > > >>>>>      partHash += row.key().hashCode();
> > > > >>>>>      partHash +=
> > > > >>>>>
> > > > >>>
> > >  Arrays.hashCode(row.value().valueBytes(grpCtx.cacheObjectContext()));
> > > > >>>>>      }
> > > > >>>>>      , are the same.
> > > > >>>>>
> > > > >>>>>      What if we'll generate same pairs
> > updateCounter-partitionHash
> > > > but
> > > > >>>>>      will compare hashes only in case counters are
the same?
> > > > >>>>>      So, for example, will ask cluster to generate
pairs for 64
> > > > >>>>>      partitions, then will find that 55 have the
same counters
> > (was
> > > > >>>>>      not updated during check) and check them.
> > > > >>>>>      The rest (64-55 = 9) partitions will be re-requested
and
> > > > >>>>>      rechecked with an additional 55.
> > > > >>>>>      This way we'll be able to check cluster is consistent
even
> > in
> > > > >>>>>      сase operations are in progress (just retrying
modified).
> > > > >>>>>
> > > > >>>>>      Risks and assumptions:
> > > > >>>>>      Using this strategy we'll check the cluster's
consistency
> > ...
> > > > >>>>>      eventually, and the check will take more time
even on an
> > idle
> > > > >>>>>      cluster.
> > > > >>>>>      In case operationsPerTimeToGeneratePartitionHashes
>
> > > > >>>>>      partitionsCount we'll definitely gain no progress.
> > > > >>>>>      But, in case of the load is not high, we'll
be able to
> check
> > > all
> > > > >>>>>      cluster.
> > > > >>>>>
> > > > >>>>>      Another hope is that we'll be able to pause/continue
scan,
> > for
> > > > >>>>>      example, we'll check 1/3 partitions today, 1/3
tomorrow,
> and
> > > in
> > > > >>>>>      three days we'll check the whole cluster.
> > > > >>>>>
> > > > >>>>>      Have I missed something?
> > > > >>>>>
> > > > >>>>>      2) Since "Idle verify" uses regular pagmem,
I assume it
> > > replaces
> > > > >>>>>      hot data with persisted.
> > > > >>>>>      So, we have to warm up the cluster after each
check.
> > > > >>>>>      Are there any chances to check without cooling
the
> cluster?
> > > > >>>>>
> > > > >>>>>      [1]
> > > > >>>>>
> > > > >>>
> > > >
> > >
> >
> https://apacheignite-tools.readme.io/docs/control-script#section-verification-of-partition-checksums
> > > >
> > >
> >
> >
> > --
> >
> > Best regards,
> > Alexei Scherbakov
> >
>


-- 

Best regards,
Alexei Scherbakov

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message