Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 5453B200D59 for ; Sat, 25 Nov 2017 14:51:20 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 52E3A160C03; Sat, 25 Nov 2017 13:51:20 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7238C160C00 for ; Sat, 25 Nov 2017 14:51:19 +0100 (CET) Received: (qmail 53511 invoked by uid 500); 25 Nov 2017 13:51:18 -0000 Mailing-List: contact notifications-help@asterixdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.apache.org Delivered-To: mailing list notifications@asterixdb.apache.org Received: (qmail 53502 invoked by uid 99); 25 Nov 2017 13:51:18 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 25 Nov 2017 13:51:18 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id C5DFC180718 for ; Sat, 25 Nov 2017 13:51:16 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 0.919 X-Spam-Level: X-Spam-Status: No, score=0.919 tagged_above=-999 required=6.31 tests=[SPF_FAIL=0.919] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id S1wzFMTNR2oh for ; Sat, 25 Nov 2017 13:51:15 +0000 (UTC) Received: from vitalstatistix.ics.uci.edu (vitalstatistix.ics.uci.edu [128.195.52.38]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 9DD1C5FCC7 for ; Sat, 25 Nov 2017 13:51:14 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by vitalstatistix.ics.uci.edu (Postfix) with ESMTP id DA82710069F; Sat, 25 Nov 2017 05:51:06 -0800 (PST) Date: Sat, 25 Nov 2017 05:51:05 -0800 From: "Murtadha Hubail (Code Review)" To: Jenkins Reply-To: mhubail@apache.org X-Gerrit-MessageType: newpatchset Subject: Change in asterixdb[master]: [NO ISSUE][STO] Adapt Structure Structure To Rebalance X-Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0 X-Gerrit-ChangeURL: X-Gerrit-Commit: b38ba5c2e38deed777a0e4744e0bf24f03e71508 In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline User-Agent: Gerrit/2.12.7 Message-Id: <20171125135106.DA82710069F@vitalstatistix.ics.uci.edu> archived-at: Sat, 25 Nov 2017 13:51:20 -0000 Hello Jenkins, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/2181 to look at the new patch set (#3). Change subject: [NO ISSUE][STO] Adapt Structure Structure To Rebalance ...................................................................... [NO ISSUE][STO] Adapt Structure Structure To Rebalance - user model changes: no - storage format changes: no - interface changes: yes -- Added IResource#setPath to use for the resource storage migration. Details: - Unify storage structure to support dataset rebalance: Old format: ./storage/partition_#/dataverse/datasetName_idx_indexName New format: ./storage/partition_#/dataverse/datasetName/rebalanaceNum/indexName - Adapt recovery and replication to new storage structure. - Add old structure -> new structure NC migration task. - Add CompatibilityUtil to ensure NC can be upgraded during NC startup. - Centralize the logic for parsing file path to its components in ResourceReference/DatasetResourceReference. - Add storage structure migration test case. - Add test case for recovery after rebalance. Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0 --- M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/RecoveryManager.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/TransactionSubsystem.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/app/nc/task/MigrateStorageResourcesTask.java M asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/NCApplication.java A asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/CompatibilityUtil.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/api/http/servlet/ConnectorApiServletTest.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/common/TestDataUtil.java A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/storage/MigrateStorageResourcesTaskTest.java M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/txn/RecoveryManagerTest.java M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/all_datasets/all_datasets.11.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/all_datasets/all_datasets.12.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/all_datasets/all_datasets.5.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/all_datasets/all_datasets.6.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataset/single_dataset.5.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataset/single_dataset.9.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataset_with_index/single_dataset_with_index.10.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataset_with_index/single_dataset_with_index.5.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataverse/single_dataverse.11.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataverse/single_dataverse.12.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataverse/single_dataverse.5.adm M asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/single_dataverse/single_dataverse.6.adm M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/dataflow/DatasetLocalResource.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/replication/IReplicaResourcesManager.java A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/DatasetResourceReference.java D asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexFileProperties.java A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/IndexPathElements.java A asterixdb/asterix-common/src/main/java/org/apache/asterix/common/storage/ResourceReference.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StorageConstants.java M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/utils/StoragePathUtil.java M asterixdb/asterix-replication/pom.xml M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationChannel.java M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/management/ReplicationManager.java M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMComponentProperties.java M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/LSMIndexFileProperties.java M asterixdb/asterix-replication/src/main/java/org/apache/asterix/replication/storage/ReplicaResourcesManager.java M asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/resource/PersistentLocalResourceRepository.java M asterixdb/asterix-transactions/src/main/java/org/apache/asterix/transaction/management/service/recovery/ReplicationCheckpointManager.java M hyracks-fullstack/hyracks/hyracks-storage-am-btree/src/main/java/org/apache/hyracks/storage/am/btree/dataflow/BTreeResource.java M hyracks-fullstack/hyracks/hyracks-storage-am-lsm-common/src/main/java/org/apache/hyracks/storage/am/lsm/common/dataflow/LsmResource.java M hyracks-fullstack/hyracks/hyracks-storage-am-rtree/src/main/java/org/apache/hyracks/storage/am/rtree/dataflow/RTreeResource.java M hyracks-fullstack/hyracks/hyracks-storage-common/src/main/java/org/apache/hyracks/storage/common/IResource.java 41 files changed, 905 insertions(+), 435 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/81/2181/3 -- To view, visit https://asterix-gerrit.ics.uci.edu/2181 To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-MessageType: newpatchset Gerrit-Change-Id: I0f968b9f493bf5aa2d49f503afe21f0d438bb7f0 Gerrit-PatchSet: 3 Gerrit-Project: asterixdb Gerrit-Branch: master Gerrit-Owner: Murtadha Hubail Gerrit-Reviewer: Jenkins