jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Jain (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (OAK-7859) S3 Bucket iterator stops too early
Date Wed, 24 Oct 2018 11:06:00 GMT

    [ https://issues.apache.org/jira/browse/OAK-7859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16662120#comment-16662120

Amit Jain commented on OAK-7859:

Thanks [~wim.symons]. Looks like those files are uploaded on schedule (every 12 hours) but
are not being deleted. While these files can be easily cleaned up and DSGC forced to update
the blob id cache, It'll be good to know why these are not being removed as they should. Could
you please change logging to DEBUG for {{org.apache.jackrabbit.oak.plugins.blob.datastore}}
when running DSGC next time and report back here.

> S3 Bucket iterator stops too early
> ----------------------------------
>                 Key: OAK-7859
>                 URL: https://issues.apache.org/jira/browse/OAK-7859
>             Project: Jackrabbit Oak
>          Issue Type: Bug
>          Components: blob-cloud
>    Affects Versions: 1.6.6
>            Reporter: Wim Symons
>            Assignee: Amit Jain
>            Priority: Critical
>              Labels: newbie, pull-request-available
>         Attachments: META-ids.txt, meta-info.txt
> Fixed a major bug in the S3 bucket iterator.
> When the returned queue of records is empty due to the fact that we get a full page of
records starting with the META/ key, the iterator stops while there is still data available
in the bucket.
> This causes problems with datastore GC, and datastore consistency checks (both online
and offline), and possibly even more.
> A little explainer. But based on a batch size of 2 instead of 1000.
> Suppose your list of S3 keys looks as follows:
>  * 1
>  * 2
>  * 3
>  * 4
>  * META/1
>  * META/2
>  * 5
>  * 6
> loadBatch would first load [1, 2], filter out no META/ keys and pass [1, 2] to the caller.
> Next time, loadBatch would load [3, 4], filter out no META/ keys and pass [3, 4] to the
> Than, loadBatch would load [META/1, META/2], filter out the META/ keys and pass [] to
the caller.
> When that happens, traversing the bucket would stop, because the returned list is empty,
even if there are many more batches to load.
> The fix checks if the returned list is empty and there are more batches available,
it would load (a) new batch(es) until there is data in the batch or there is no more batch
> We are currently running Oak 1.6.6 on AEM, but as the bug is still in trunk,
all previous versions of Oak are affected as well.
> I provided 2 pull requests: one for trunk ([https://github.com/apache/jackrabbit-oak/pull/103)] and
one for the 1.6 branch ([https://github.com/apache/jackrabbit-oak/pull/104).]
> CI failed on [https://github.com/apache/jackrabbit-oak/pull/103,] but I don't think
it's related to my changes.
> For the record, the patch works as I was able to successfully test this on our production
repository using oak-run --id. With version 1.6.6 it reported 800k items, with my patched
version, it reported 1.8m items. (As our META/ nodes are listed somewhere half-way through.)

This message was sent by Atlassian JIRA

View raw message