asterixdb-notifications mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Murtadha Hubail (Code Review)" <do-not-re...@asterixdb.incubator.apache.org>
Subject Change in asterixdb[master]: [NO ISSUE][STO] Adapt Storagee Structure To Rebalance
Date Sat, 25 Nov 2017 13:52:27 GMT
Hello Jenkins,

I'd like you to reexamine a change.  Please visit

    https://asterix-gerrit.ics.uci.edu/2181

to look at the new patch set (#4).

Change subject: [NO ISSUE][STO] Adapt Storagee Structure To Rebalance
......................................................................

[NO ISSUE][STO] Adapt Storagee Structure To Rebalance

- user model changes: no
- storage format changes: no
- interface changes: yes
    -- Added IResource#setPath to use for the resource
       storage migration.

Details:
- Unify storage structure to support dataset rebalance:
  Old format:
  ./storage/partition_#/dataverse/datasetName_idx_indexName
  New format:
  ./storage/partition_#/dataverse/datasetName/rebalanaceNum/indexName
- Adapt recovery and replication to new storage structure.
- Add old structure -> new structure NC migration task.
- Add CompatibilityUtil to ensure NC can be upgraded during
  NC startup.
- Centralize the logic for parsing file path to its components in
  ResourceReference/DatasetResourceReference.
- Add storage structure migration test case.
- Add test case for recovery after rebalance.

Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
---
M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/TransactionSubsystem.java
A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/task/MigrateStorageResourcesTask.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java
A asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/CompatibilityUtil.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/api/http/servlet/ConnectorApiServletTest.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/common/TestDataUtil.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/storage/MigrateStorageResourcesTaskTest.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/txn/RecoveryManagerTest.java
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/all_datasets/all_datasets.11.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/all_datasets/all_datasets.12.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/all_datasets/all_datasets.5.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/all_datasets/all_datasets.6.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataset/single_dataset.5.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataset/single_dataset.9.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataset_with_index/single_dataset_with_index.10.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataset_with_index/single_dataset_with_index.5.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataverse/single_dataverse.11.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataverse/single_dataverse.12.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataverse/single_dataverse.5.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataverse/single_dataverse.6.adm
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/dataflow/DatasetLocalResource.java
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java
A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/DatasetResourceReference.java
D asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexFileProperties.java
A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexPathElements.java
A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/ResourceReference.java
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java
M asterixdb/asterix-replication/pom.xml
M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java
M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java
M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMComponentProperties.java
M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java
M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java
M asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
M asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java
M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/dataflow/BTreeResource.java
M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/dataflow/LsmResource.java
M hyracks-fullstack/hyracks/hyracks-storage-am-rtree/src/main/java/org/apache/hyracks/storage/am/rtree/dataflow/RTreeResource.java
M hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/IResource.java
41 files changed, 905 insertions(+), 435 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/81/2181/4
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/2181
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0
Gerrit-PatchSet: 4
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <mhubail@apache.org>
Gerrit-Reviewer: Jenkins <jenkins@fulliautomatix.ics.uci.edu>

Mime
View raw message