lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lucene/Solr QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SOLR-9961) RestoreCore needs the option to download files in parallel.
Date Sun, 30 Jun 2019 11:20:00 GMT

    [ https://issues.apache.org/jira/browse/SOLR-9961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16875750#comment-16875750
] 

Lucene/Solr QA commented on SOLR-9961:
--------------------------------------

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m  1s{color} |
{color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 54s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 54s{color} | {color:green}
the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} | {color:green}  1m 54s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} | {color:green}  1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} | {color:green}
 1m 54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 33m  5s{color} | {color:green}
core in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m  7s{color} | {color:black}
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-9961 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12973241/SOLR-9961.patch
|
| Optional Tests |  compile  javac  unit  ratsources  checkforbiddenapis  validatesourcepatterns
 |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon Sep 24 17:14:57
UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality | /home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 2fdb4dd |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | LTS |
|  Test Results | https://builds.apache.org/job/PreCommit-SOLR-Build/473/testReport/ |
| modules | C: solr/core U: solr/core |
| Console output | https://builds.apache.org/job/PreCommit-SOLR-Build/473/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> RestoreCore needs the option to download files in parallel.
> -----------------------------------------------------------
>
>                 Key: SOLR-9961
>                 URL: https://issues.apache.org/jira/browse/SOLR-9961
>             Project: Solr
>          Issue Type: Improvement
>          Components: Backup/Restore
>    Affects Versions: 6.2.1
>            Reporter: Timothy Potter
>            Priority: Major
>         Attachments: SOLR-9961.patch, SOLR-9961.patch, SOLR-9961.patch, SOLR-9961.patch
>
>
> My backup to cloud storage (Google cloud storage in this case, but I think this is a
general problem) takes 8 minutes ... the restore of the same core takes hours. The restore
loop in RestoreCore is serial and doesn't allow me to parallelize the expensive part of this
operation (the IO from the remote cloud storage service). We need the option to parallelize
the download (like distcp). 
> Also, I tried downloading the same directory using gsutil and it was very fast, like
2 minutes. So I know it's not the pipe that's limiting perf here.
> Here's a very rough patch that does the parallelization. We may also want to consider
a two-step approach: 1) download in parallel to a temp dir, 2) perform all the of the checksum
validation against the local temp dir. That will save round trips to the remote cloud storage.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message