jackrabbit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shashank Gupta (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (JCR-3731) Multi-threaded migration of binary files from FileSystem to S3 datastore
Date Thu, 06 Mar 2014 05:35:42 GMT

     [ https://issues.apache.org/jira/browse/JCR-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Shashank Gupta updated JCR-3731:

    Attachment: JCR-3651-svn-updated.patch

Attached patched. 
Created thread pool to migrate files from file system to S3. For 
570 MB initial data, time reduced to 70s from 307s. 
h5. Migration using 1 thread:
18.12.2013 05:08:36.877 *INFO* [FelixStartLevel] 
com.day.crx.cloud.s3.ds.CachingDataStore Uploaded:  {3875}/{3875} files, 
{570535403}/{570535403} size data, time taken {307} sec

> Multi-threaded migration of binary files from FileSystem to S3 datastore 
> -------------------------------------------------------------------------
>                 Key: JCR-3731
>                 URL: https://issues.apache.org/jira/browse/JCR-3731
>             Project: Jackrabbit Content Repository
>          Issue Type: Sub-task
>          Components: jackrabbit-core
>    Affects Versions: 2.7.4
>            Reporter: Shashank Gupta
>             Fix For: 2.7.5
>         Attachments: JCR-3651-svn-updated.patch
> As per today, when we switch repository from FileDataStore to S3DataStore all binary
files are migrated from local file system to S3Datastore. As per today this process is single
threaded and takes lot of time. For e.g. for 1G intial content, it takes around 5 min to migrated
from ec2 instance to S3.
> It can be made faster by migrating content in multi-threaded environment.

This message was sent by Atlassian JIRA

View raw message