jackrabbit-oak-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chetan Mehrotra (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (OAK-6246) Support for out of band indexing with read only access to NodeStore
Date Fri, 02 Jun 2017 11:53:05 GMT

    [ https://issues.apache.org/jira/browse/OAK-6246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034551#comment-16034551
] 

Chetan Mehrotra edited comment on OAK-6246 at 6/2/17 11:52 AM:
---------------------------------------------------------------

Implemented a script to measure such a traversal [here|https://github.com/chetanmeh/oak-console-scripts/blob/master/src/main/groovy/traversal/traversalRateEstimator.groovy].
To run it use

{noformat}
java -jar oak-run-1.8-SNAPSHOT.jar console --metrics mongodb://server:port/db ":load https://raw.githubusercontent.com/chetanmeh/oak-console-scripts/master/src/main/groovy/traversal/traversalRateEstimator.groovy"
{noformat}

With this I am observing traversal rate of 110k nodes/sec compared to 1.4 nodes/sec with NodeStore
based traversal :)


was (Author: chetanm):
Implemented a script to measure such a traversal [here|https://github.com/chetanmeh/oak-console-scripts/blob/master/src/main/groovy/traversal/traversalRateEstimator.groovy].
To run it use

{noformat}
java -jar oak-run-1.8-SNAPSHOT.jar console --metrics mongodb://server:port/db ":load https://raw.githubusercontent.com/chetanmeh/oak-console-scripts/master/src/main/groovy/traversal/traversalRateEstimator.groovy"
{noformat}

> Support for out of band indexing with read only access to NodeStore
> -------------------------------------------------------------------
>
>                 Key: OAK-6246
>                 URL: https://issues.apache.org/jira/browse/OAK-6246
>             Project: Jackrabbit Oak
>          Issue Type: New Feature
>          Components: run
>            Reporter: Chetan Mehrotra
>            Assignee: Chetan Mehrotra
>             Fix For: 1.8
>
>
> Provide support for out of band indexing where oak-run is connected in read only mode
with NodeStore and indexes are stored on file system. These are then imported back by target
system.
> Had a discussion with [~catholicon] and following flow was determined
> # Admin would create provision a checkpoint via CheckpointMBean
> # oak-run index is connected to NodeStore in read only mode and passed with
> #* checkpoint from previous step
> #* list of indexes which need to be reindexed
> # oak-run index logic would then proceed with reindexing. However the created index data
would be stored locally. This would make use of 
> #* DirectoryFactory - OAK-6243
> #* Copy-on-write nodestore approach as being used in OAK-6220
> # Once indexing is completed it would dump all index to an output folder with some metadata
> # Then admin can copy this index data and use an MBean on the target setup to "import"
it back. This import would need to
> #* Pause the current async indexers
> #* Import the external index files
> #* Bring the external indexer upto date to there respective lanes checkpoint
> #* Resume the async indexer
> The benefit of this approach is that 
> # We only need to backport the import logic. Rest all can be implemented in trunk and
need not be backported. 
> # Using read-only mode allow oak-run from trunk to be safely connected to any of the
old versions



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message