hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-11185) Parallelize Snapshot operations
Date Wed, 20 Aug 2014 08:30:28 GMT

    [ https://issues.apache.org/jira/browse/HBASE-11185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14103608#comment-14103608
] 

Hudson commented on HBASE-11185:
--------------------------------

FAILURE: Integrated in HBase-0.98 #460 (See [https://builds.apache.org/job/HBase-0.98/460/])
HBASE-11742 (Backport HBASE-7987 and HBASE-11185 to 0.98) (Esteban Gutierrez and Matteo Bertozzi)
(apurtell: rev 2e978a165481c45c6a3d97b185c797a0425ddc44)
* hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MapReduceProtos.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV2.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifestV1.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/RestoreSnapshotHandler.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotTask.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapred/TableSnapshotInputFormat.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TableInfoCopyTask.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotLogSplitter.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ReferenceServerWALsTask.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/DisabledTableSnapshotHandler.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestSnapshotFromMaster.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/io/Reference.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotDescriptionUtils.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/TakeSnapshotHandler.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/CopyRecoveredEditsTask.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableSnapshotInputFormat.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreSnapshotHelper.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotLogSplitter.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotFileCache.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormat.java
* hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/SnapshotProtos.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSVisitor.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotInfo.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/CloneSnapshotHandler.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotFromClient.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/EnabledTableSnapshotHandler.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/master/snapshot/TestSnapshotFileCache.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/client/TableSnapshotScanner.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotLogCleaner.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatImpl.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java
* hbase-protocol/src/main/protobuf/Snapshot.proto
* hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestReferenceRegionHFilesTask.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestCopyRecoveredEditsTask.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotHFileCleaner.java
* hbase-protocol/src/main/protobuf/MapReduce.proto
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/TakeSnapshotUtils.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestSnapshotTask.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestWALReferenceTask.java
* hbase-common/src/main/resources/hbase-default.xml
* hbase-server/src/main/java/org/apache/hadoop/hbase/procedure/ProcedureManager.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/MasterSnapshotVerifier.java
* hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotDescriptionUtils.java
* hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ReferenceRegionHFilesTask.java


> Parallelize Snapshot operations
> -------------------------------
>
>                 Key: HBASE-11185
>                 URL: https://issues.apache.org/jira/browse/HBASE-11185
>             Project: HBase
>          Issue Type: Bug
>          Components: snapshots
>    Affects Versions: 0.99.0
>            Reporter: Matteo Bertozzi
>            Assignee: Matteo Bertozzi
>            Priority: Minor
>             Fix For: 0.99.0
>
>         Attachments: HBASE-11185-v0.patch
>
>
> when SnapshotInfo or snapshot verification is executed against a remote path, it may
takes a while since all the code is mainly composed by sequential calls to the fs.
> This patch will parallelize all the snapshot operations using a thread pool to dispatch
requests. The size of the pool is tunable by using  "hbase.snapshot.thread.pool.max"



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message