jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Jain (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (OAK-8170) oak-run datastorecheck and online consistency check falsely report missing blobs
Date Wed, 27 Mar 2019 09:16:00 GMT

    [ https://issues.apache.org/jira/browse/OAK-8170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16802574#comment-16802574

Amit Jain commented on OAK-8170:

{quote}So my question still stands: do (you think) we actually have missing blobs in our repository
(from an old node revision)?
Well the blob is missing yes but that blob is not actually needed as it is not being used.
Not sure as to why this has happened, maybe it's because of the ActiveDeletionLuceneBinaries
where it proactively deletes lucene binaries not needed.

As regards to the order
t1 - backup repository using AEM online backup

t2 - check consistency of backup segment store (oak-run check)

t3 - check consistency of backup segment store vs datastore (oak-run datastorecheck)

t4 - run online tail/full compaction

t5 - run DSGC on a shared S3 datastore
The t3. check consistency operation makes sense to run after t5. DSGC. Any likely failure
of DSGC can potentially cause missing blobs where systems are misconfigured and the objective
would be to catch that right away.

> oak-run datastorecheck and online consistency check falsely report missing blobs
> --------------------------------------------------------------------------------
>                 Key: OAK-8170
>                 URL: https://issues.apache.org/jira/browse/OAK-8170
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: segment-tar
>    Affects Versions: 1.8.9
>            Reporter: Wim Symons
>            Priority: Major
>         Attachments: output.txt
> Hi,
> We found that oak-run datastorecheck falsely reports missing blobs when running datastorecheck
without the --verbose option.
> Even the online datastore consistency check falsely reports the same missing blobs.
> This is related due to the fact that the standard blob reference collector in oak-run
datastorecheck looks at *all* compaction generations in the segment store instead of only
the last one.
> After running an offline compaction, and thus keeping only 1 generation, the correct
number of blob references and missing blobs is reported by oak-run datastorecheck.
> The bug on the 1.8 branch comes from org.apache.jackrabbit.oak.plugins.blob.BlobReferenceRetriever#collectReferences
(line 429) and by following that you arrive at org.apache.jackrabbit.oak.segment.file.FileStore#tarFiles
(line 1013) stating:
> tarFiles.collectBlobReferences(collector,
>  newOldReclaimer(lastCompactionType, getGcGeneration(), gcOptions.getRetainedGenerations()));
> I'm not familiar enough with this source code, so I won't attempt adding a patch.
> I did double-check trunk and saw the same line of code there: org.apache.jackrabbit.oak.segment.file.GarbageCollector#collectBlobReferences
(line 324).
> I attached a text file with the outputs of the commands I ran.
> We currently use Oak 1.8.9 using AEM and oak-blob-cloud 1.8.9 from the 1.8.3
AEM S3 connector.
> Regards
> Wim

This message was sent by Atlassian JIRA

View raw message