jackrabbit-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shashank Gupta (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (JCR-3731) Multi-threaded migration of binary files from FileSystem to S3 datastore
Date Thu, 06 Mar 2014 05:37:42 GMT

    [ https://issues.apache.org/jira/browse/JCR-3731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13922040#comment-13922040
] 

Shashank Gupta edited comment on JCR-3731 at 3/6/14 5:36 AM:
-------------------------------------------------------------

Attached patched. 
Created thread pool to migrate files from file system to S3. For 
570 MB initial data, time reduced to 70s from 307s. 
Result:
h5. Migration using 1 thread:
{code}
18.12.2013 05:08:36.877 *INFO* [FelixStartLevel] 
org.apache.jackrabbit.core.data.CachingDataStore Uploaded:  {3875}/{3875} files, 
{570535403}/{570535403} size data, time taken {307} sec
{code}

h5. Migration using 10 threads:
{code}
18.12.2013 04:56:52.176 *INFO* [FelixStartLevel] 
org.apache.jackrabbit.core.data.CachingDataStore Uploaded:  {3875}/{3875} files, 
{570535403}/{570535403} size data, time taken {70} sec
{code}



was (Author: shgupta):
Attached patched. 
Created thread pool to migrate files from file system to S3. For 
570 MB initial data, time reduced to 70s from 307s. 
Result:
h5. Migration using 1 thread:
{code}
18.12.2013 05:08:36.877 *INFO* [FelixStartLevel] 
com.day.crx.cloud.s3.ds.CachingDataStore Uploaded:  {3875}/{3875} files, 
{570535403}/{570535403} size data, time taken {307} sec
{code}


> Multi-threaded migration of binary files from FileSystem to S3 datastore 
> -------------------------------------------------------------------------
>
>                 Key: JCR-3731
>                 URL: https://issues.apache.org/jira/browse/JCR-3731
>             Project: Jackrabbit Content Repository
>          Issue Type: Sub-task
>          Components: jackrabbit-core
>    Affects Versions: 2.7.4
>            Reporter: Shashank Gupta
>             Fix For: 2.7.5
>
>         Attachments: JCR-3651-svn-updated.patch
>
>
> As per today, when we switch repository from FileDataStore to S3DataStore all binary
files are migrated from local file system to S3Datastore. As per today this process is single
threaded and takes lot of time. For e.g. for 1G intial content, it takes around 5 min to migrated
from ec2 instance to S3.
> It can be made faster by migrating content in multi-threaded environment.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message