Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 8EBC6200C82 for ; Sat, 27 May 2017 21:23:55 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 8D727160BB8; Sat, 27 May 2017 19:23:55 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 38811160BDA for ; Sat, 27 May 2017 21:23:53 +0200 (CEST) Received: (qmail 46475 invoked by uid 500); 27 May 2017 19:23:52 -0000 Mailing-List: contact commits-help@asterixdb.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@asterixdb.apache.org Delivered-To: mailing list commits@asterixdb.apache.org Received: (qmail 46417 invoked by uid 99); 27 May 2017 19:23:52 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 27 May 2017 19:23:52 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 38485E117B; Sat, 27 May 2017 19:23:52 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: buyingyi@apache.org To: commits@asterixdb.apache.org Date: Sat, 27 May 2017 19:23:54 -0000 Message-Id: In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [3/3] asterixdb git commit: Add a dataset rebalance REST API. archived-at: Sat, 27 May 2017 19:23:55 -0000 Add a dataset rebalance REST API. - Failures during rebalance are not handled; - Indexes are not built for the rebalance target. Change-Id: Ibda35252031fc4940972f0f19bbf796cadfa53d6 Reviewed-on: https://asterix-gerrit.ics.uci.edu/1768 Tested-by: Jenkins BAD: Jenkins Integration-Tests: Jenkins Reviewed-by: Michael Blow Project: http://git-wip-us.apache.org/repos/asf/asterixdb/repo Commit: http://git-wip-us.apache.org/repos/asf/asterixdb/commit/87805197 Tree: http://git-wip-us.apache.org/repos/asf/asterixdb/tree/87805197 Diff: http://git-wip-us.apache.org/repos/asf/asterixdb/diff/87805197 Branch: refs/heads/master Commit: 878051979401f95be6e380ff48d1dcf89894b22b Parents: 75d32e5 Author: Yingyi Bu Authored: Sat May 27 10:06:15 2017 -0700 Committer: Yingyi Bu Committed: Sat May 27 12:23:10 2017 -0700 ---------------------------------------------------------------------- .../operators/physical/CommitPOperator.java | 11 +- .../api/http/server/ConnectorApiServlet.java | 3 +- .../api/http/server/RebalanceApiServlet.java | 105 +++++++ .../asterix/app/translator/QueryTranslator.java | 5 +- .../hyracks/bootstrap/CCApplication.java | 4 + .../org/apache/asterix/utils/DataverseUtil.java | 7 +- .../org/apache/asterix/utils/RebalanceUtil.java | 294 +++++++++++++++++++ .../app/bootstrap/TestNodeController.java | 2 +- .../asterix/test/runtime/RebalanceTest.java | 64 ++++ .../duplicate_location.1.ddl.sqlpp | 47 +++ .../duplicate_location.2.update.sqlpp | 23 ++ .../duplicate_location.3.post.http | 20 ++ .../duplicate_location.4.query.sqlpp | 22 ++ .../duplicate_location.5.query.sqlpp | 22 ++ .../empty_location/empty_location.1.ddl.sqlpp | 47 +++ .../empty_location.2.update.sqlpp | 23 ++ .../empty_location/empty_location.3.post.http | 20 ++ .../empty_location/empty_location.4.query.sqlpp | 22 ++ .../identical_location.1.ddl.sqlpp | 47 +++ .../identical_location.2.update.sqlpp | 23 ++ .../identical_location.3.post.http | 20 ++ .../identical_location.4.query.sqlpp | 22 ++ .../nonexist_dataset.1.post.http | 20 ++ .../single_dataset/single_dataset.1.ddl.sqlpp | 47 +++ .../single_dataset.10.query.sqlpp | 22 ++ .../single_dataset.11.query.sqlpp | 22 ++ .../single_dataset.2.update.sqlpp | 23 ++ .../single_dataset/single_dataset.3.query.sqlpp | 22 ++ .../single_dataset/single_dataset.4.post.http | 20 ++ .../single_dataset/single_dataset.5.get.http | 20 ++ .../single_dataset/single_dataset.6.query.sqlpp | 22 ++ .../single_dataset/single_dataset.7.query.sqlpp | 22 ++ .../single_dataset/single_dataset.8.post.http | 20 ++ .../single_dataset/single_dataset.9.get.http | 20 ++ .../src/test/resources/runtimets/rebalance.xml | 50 ++++ .../duplicate_location/duplicate_location.3.adm | 1 + .../duplicate_location/duplicate_location.4.adm | 1 + .../duplicate_location/duplicate_location.5.adm | 1 + .../empty_location/empty_location.3.adm | 1 + .../identical_location/identical_location.3.adm | 1 + .../identical_location/identical_location.4.adm | 1 + .../nonexist_dataset/nonexist_dataset.1.adm | 1 + .../single_dataset/single_dataset.10.adm | 1 + .../single_dataset/single_dataset.11.adm | 1 + .../single_dataset/single_dataset.3.adm | 1 + .../single_dataset/single_dataset.4.adm | 1 + .../single_dataset/single_dataset.5.adm | 1 + .../single_dataset/single_dataset.6.adm | 1 + .../single_dataset/single_dataset.7.adm | 1 + .../single_dataset/single_dataset.8.adm | 1 + .../single_dataset/single_dataset.9.adm | 1 + .../asterix/common/exceptions/ErrorCode.java | 1 + .../apache/asterix/common/utils/Servlets.java | 1 + .../asterix/common/utils/StoragePathUtil.java | 11 +- .../main/resources/asx_errormsg/en.properties | 1 + .../apache/asterix/metadata/MetadataCache.java | 8 +- .../metadata/bootstrap/MetadataBootstrap.java | 4 - .../metadata/bootstrap/MetadataIndex.java | 6 +- .../metadata/declared/MetadataManagerUtil.java | 5 + .../metadata/declared/MetadataProvider.java | 113 ++----- .../asterix/metadata/entities/Dataset.java | 184 +++++++++--- .../apache/asterix/metadata/entities/Index.java | 1 - .../DatasetTupleTranslator.java | 30 +- .../asterix/metadata/utils/DatasetUtil.java | 234 +++++++++++++-- .../asterix/metadata/utils/IndexUtil.java | 31 ++ .../utils/SecondaryBTreeOperationsHelper.java | 11 +- .../utils/SecondaryIndexOperationsHelper.java | 67 +---- .../SecondaryInvertedIndexOperationsHelper.java | 11 +- .../utils/SecondaryRTreeOperationsHelper.java | 10 +- .../utils/SplitsAndConstraintsUtil.java | 14 +- .../BinaryHashFunctionFactoryProvider.java | 14 - .../PersistentLocalResourceRepository.java | 3 +- .../service/transaction/DatasetIdFactory.java | 26 +- .../constraints/PartitionConstraintHelper.java | 11 +- .../hyracks/api/exceptions/ErrorCode.java | 1 + .../src/main/resources/errormsg/en.properties | 1 + 76 files changed, 1681 insertions(+), 317 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java index 20c69c4..2dd4c3d 100644 --- a/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java +++ b/asterixdb/asterix-algebra/src/main/java/org/apache/asterix/algebra/operators/physical/CommitPOperator.java @@ -88,15 +88,8 @@ public class CommitPOperator extends AbstractPhysicalOperator { int[] primaryKeyFields = JobGenHelper.variablesToFieldIndexes(primaryKeyLogicalVars, inputSchemas[0]); //get dataset splits - FileSplit[] splitsForDataset = metadataProvider.splitsForDataset(metadataProvider.getMetadataTxnContext(), - dataset.getDataverseName(), dataset.getDatasetName(), dataset.getDatasetName(), - metadataProvider.isTemporaryDatasetWriteJob()); - int[] datasetPartitions = new int[splitsForDataset.length]; - for (int i = 0; i < splitsForDataset.length; i++) { - datasetPartitions[i] = i; - } - IPushRuntimeFactory runtime = dataset.getCommitRuntimeFactory(jobId, primaryKeyFields, metadataProvider, - datasetPartitions, isSink); + IPushRuntimeFactory runtime = dataset.getCommitRuntimeFactory(metadataProvider, jobId, primaryKeyFields, + isSink); builder.contributeMicroOperator(op, runtime, recDesc); ILogicalOperator src = op.getInputs().get(0).getValue(); builder.contributeGraphEdge(src, 0, op, 0); http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/ConnectorApiServlet.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/ConnectorApiServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/ConnectorApiServlet.java index 421ee0e..396665e 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/ConnectorApiServlet.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/ConnectorApiServlet.java @@ -105,8 +105,7 @@ public class ConnectorApiServlet extends AbstractServlet { return; } boolean temp = dataset.getDatasetDetails().isTemp(); - FileSplit[] fileSplits = - metadataProvider.splitsForDataset(mdTxnCtx, dataverseName, datasetName, datasetName, temp); + FileSplit[] fileSplits = metadataProvider.splitsForIndex(mdTxnCtx, dataset, datasetName); ARecordType recordType = (ARecordType) metadataProvider.findType(dataset.getItemTypeDataverseName(), dataset.getItemTypeName()); List> primaryKeys = dataset.getPrimaryKeys(); http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/RebalanceApiServlet.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/RebalanceApiServlet.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/RebalanceApiServlet.java new file mode 100644 index 0000000..0d525c8 --- /dev/null +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/api/http/server/RebalanceApiServlet.java @@ -0,0 +1,105 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.asterix.api.http.server; + +import static org.apache.asterix.api.http.servlet.ServletConstants.HYRACKS_CONNECTION_ATTR; + +import java.io.PrintWriter; +import java.util.Arrays; +import java.util.LinkedHashSet; +import java.util.concurrent.ConcurrentMap; +import java.util.logging.Level; +import java.util.logging.Logger; + +import org.apache.asterix.common.dataflow.ICcApplicationContext; +import org.apache.asterix.file.StorageComponentProvider; +import org.apache.asterix.metadata.declared.MetadataProvider; +import org.apache.asterix.utils.RebalanceUtil; +import org.apache.commons.lang3.StringUtils; +import org.apache.hyracks.api.client.IHyracksClientConnection; +import org.apache.hyracks.http.api.IServletRequest; +import org.apache.hyracks.http.api.IServletResponse; +import org.apache.hyracks.http.server.AbstractServlet; +import org.apache.hyracks.http.server.utils.HttpUtil; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.node.ObjectNode; + +import io.netty.handler.codec.http.HttpResponseStatus; + +/** + * This servlet processes dataset rebalance requests. + */ +public class RebalanceApiServlet extends AbstractServlet { + private static final Logger LOGGER = Logger.getLogger(RebalanceApiServlet.class.getName()); + private ICcApplicationContext appCtx; + + public RebalanceApiServlet(ConcurrentMap ctx, String[] paths, ICcApplicationContext appCtx) { + super(ctx, paths); + this.appCtx = appCtx; + } + + @Override + protected void post(IServletRequest request, IServletResponse response) { + PrintWriter out = response.writer(); + ObjectMapper om = new ObjectMapper(); + ObjectNode jsonResponse = om.createObjectNode(); + try { + // Sets the content type. + HttpUtil.setContentType(response, HttpUtil.ContentType.APPLICATION_JSON, HttpUtil.Encoding.UTF8); + + // Gets dataverse, dataset, and target nodes for rebalance. + String dataverseName = request.getParameter("dataverseName"); + String datasetName = request.getParameter("datasetName"); + String nodes = request.getParameter("nodes"); + if (dataverseName == null || datasetName == null || nodes == null) { + sendResponse(out, jsonResponse, response, HttpResponseStatus.BAD_REQUEST, + "dataverseName, datasetName, or nodes is not given"); + return; + } + String nodesString = StringUtils.strip(nodes, "\"'").trim(); + String[] targetNodes = nodesString.split(","); + if ("".equals(nodesString)) { + sendResponse(out, jsonResponse, response, HttpResponseStatus.BAD_REQUEST, + "target nodes should not be empty"); + return; + } + + // Rebalances a given dataset from its current locations to the target nodes. + IHyracksClientConnection hcc = (IHyracksClientConnection) ctx.get(HYRACKS_CONNECTION_ATTR); + MetadataProvider metadataProvider = new MetadataProvider(appCtx, null, new StorageComponentProvider()); + RebalanceUtil.rebalance(dataverseName, datasetName, new LinkedHashSet<>(Arrays.asList(targetNodes)), + metadataProvider, hcc); + + // Sends response. + sendResponse(out, jsonResponse, response, HttpResponseStatus.OK, "successful"); + } catch (Exception e) { + sendResponse(out, jsonResponse, response, HttpResponseStatus.INTERNAL_SERVER_ERROR, e.getMessage()); + LOGGER.log(Level.WARNING, e.getMessage(), e); + } + } + + // Sends HTTP response to the request client. + private void sendResponse(PrintWriter out, ObjectNode jsonResponse, IServletResponse response, + HttpResponseStatus status, String message) { + jsonResponse.put("results", message); + response.setStatus(status); + out.write(jsonResponse.toString()); + } +} http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java index f64812e..8ea34fd 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java @@ -613,9 +613,7 @@ public class QueryTranslator extends AbstractLangTranslator implements IStatemen MetadataUtil.PENDING_ADD_OP); MetadataManager.INSTANCE.addDataset(metadataProvider.getMetadataTxnContext(), dataset); if (dd.getDatasetType() == DatasetType.INTERNAL) { - Dataverse dataverse = - MetadataManager.INSTANCE.getDataverse(metadataProvider.getMetadataTxnContext(), dataverseName); - JobSpecification jobSpec = DatasetUtil.createDatasetJobSpec(dataverse, datasetName, metadataProvider); + JobSpecification jobSpec = DatasetUtil.createDatasetJobSpec(dataset, metadataProvider); // #. make metadataTxn commit before calling runJob. MetadataManager.INSTANCE.commitTransaction(mdTxnCtx); @@ -2446,7 +2444,6 @@ public class QueryTranslator extends AbstractLangTranslator implements IStatemen } protected void handleCreateNodeGroupStatement(MetadataProvider metadataProvider, Statement stmt) throws Exception { - NodegroupDecl stmtCreateNodegroup = (NodegroupDecl) stmt; String ngName = stmtCreateNodegroup.getNodegroupName().getValue(); http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java index 57cc340..61d93f5 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/hyracks/bootstrap/CCApplication.java @@ -46,6 +46,7 @@ import org.apache.asterix.api.http.server.QueryResultApiServlet; import org.apache.asterix.api.http.server.QueryServiceServlet; import org.apache.asterix.api.http.server.QueryStatusApiServlet; import org.apache.asterix.api.http.server.QueryWebInterfaceServlet; +import org.apache.asterix.api.http.server.RebalanceApiServlet; import org.apache.asterix.api.http.server.ShutdownApiServlet; import org.apache.asterix.api.http.server.UpdateApiServlet; import org.apache.asterix.api.http.server.VersionApiServlet; @@ -226,6 +227,7 @@ public class CCApplication extends BaseCCApplication { addServlet(jsonAPIServer, Servlets.SHUTDOWN); addServlet(jsonAPIServer, Servlets.VERSION); addServlet(jsonAPIServer, Servlets.CLUSTER_STATE); + addServlet(jsonAPIServer, Servlets.REBALANCE); addServlet(jsonAPIServer, Servlets.CLUSTER_STATE_NODE_DETAIL); // must not precede add of CLUSTER_STATE addServlet(jsonAPIServer, Servlets.CLUSTER_STATE_CC_DETAIL); // must not precede add of CLUSTER_STATE addServlet(jsonAPIServer, Servlets.DIAGNOSTICS); @@ -283,6 +285,8 @@ public class CCApplication extends BaseCCApplication { componentProvider); case Servlets.CONNECTOR: return new ConnectorApiServlet(ctx, paths, appCtx); + case Servlets.REBALANCE: + return new RebalanceApiServlet(ctx, paths, appCtx); case Servlets.SHUTDOWN: return new ShutdownApiServlet(ctx, paths); case Servlets.VERSION: http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/DataverseUtil.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/DataverseUtil.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/DataverseUtil.java index 039933a..73c841e 100644 --- a/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/DataverseUtil.java +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/DataverseUtil.java @@ -35,10 +35,9 @@ public class DataverseUtil { public static JobSpecification dropDataverseJobSpec(Dataverse dataverse, MetadataProvider metadata) { JobSpecification jobSpec = RuntimeUtils.createJobSpecification(metadata.getApplicationContext()); - Pair splitsAndConstraint = - metadata.splitAndConstraints(dataverse.getDataverseName()); - FileRemoveOperatorDescriptor frod = - new FileRemoveOperatorDescriptor(jobSpec, splitsAndConstraint.first, false); + Pair splitsAndConstraint = metadata + .splitAndConstraints(dataverse.getDataverseName()); + FileRemoveOperatorDescriptor frod = new FileRemoveOperatorDescriptor(jobSpec, splitsAndConstraint.first, false); AlgebricksPartitionConstraintHelper.setPartitionConstraintInJobSpec(jobSpec, frod, splitsAndConstraint.second); jobSpec.addRoot(frod); return jobSpec; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/RebalanceUtil.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/RebalanceUtil.java b/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/RebalanceUtil.java new file mode 100644 index 0000000..514a837 --- /dev/null +++ b/asterixdb/asterix-app/src/main/java/org/apache/asterix/utils/RebalanceUtil.java @@ -0,0 +1,294 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.asterix.utils; + +import static org.apache.asterix.app.translator.QueryTranslator.abort; + +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Set; +import java.util.UUID; +import java.util.stream.IntStream; + +import org.apache.asterix.common.exceptions.AsterixException; +import org.apache.asterix.common.transactions.JobId; +import org.apache.asterix.common.utils.JobUtils; +import org.apache.asterix.dataflow.data.nontagged.MissingWriterFactory; +import org.apache.asterix.metadata.MetadataManager; +import org.apache.asterix.metadata.MetadataTransactionContext; +import org.apache.asterix.metadata.declared.MetadataManagerUtil; +import org.apache.asterix.metadata.declared.MetadataProvider; +import org.apache.asterix.metadata.entities.Dataset; +import org.apache.asterix.metadata.entities.Index; +import org.apache.asterix.metadata.entities.NodeGroup; +import org.apache.asterix.metadata.lock.LockList; +import org.apache.asterix.metadata.lock.MetadataLockManager; +import org.apache.asterix.metadata.utils.DatasetUtil; +import org.apache.asterix.metadata.utils.IndexUtil; +import org.apache.asterix.metadata.utils.MetadataConstants; +import org.apache.asterix.runtime.job.listener.JobEventListenerFactory; +import org.apache.asterix.transaction.management.service.transaction.JobIdFactory; +import org.apache.hyracks.algebricks.common.constraints.AlgebricksPartitionConstraint; +import org.apache.hyracks.algebricks.common.constraints.AlgebricksPartitionConstraintHelper; +import org.apache.hyracks.algebricks.common.exceptions.AlgebricksException; +import org.apache.hyracks.algebricks.common.utils.Pair; +import org.apache.hyracks.algebricks.runtime.base.IPushRuntimeFactory; +import org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor; +import org.apache.hyracks.api.client.IHyracksClientConnection; +import org.apache.hyracks.api.dataflow.IConnectorDescriptor; +import org.apache.hyracks.api.dataflow.IOperatorDescriptor; +import org.apache.hyracks.api.dataflow.value.RecordDescriptor; +import org.apache.hyracks.api.job.JobSpecification; +import org.apache.hyracks.dataflow.common.data.partition.FieldHashPartitionComputerFactory; +import org.apache.hyracks.dataflow.std.connectors.MToNPartitioningConnectorDescriptor; +import org.apache.hyracks.dataflow.std.connectors.OneToOneConnectorDescriptor; + +/** + * A utility class for the rebalance operation. + */ +public class RebalanceUtil { + + private RebalanceUtil() { + + } + + /** + * Rebalances an existing dataset to a list of target nodes. + * + * @param dataverseName, + * the dataverse name. + * @param datasetName, + * the dataset name. + * @param targetNcNames, + * the list of target nodes. + * @param metadataProvider, + * the metadata provider. + * @param hcc, + * the reusable hyracks connection. + * @throws Exception + */ + public static void rebalance(String dataverseName, String datasetName, Set targetNcNames, + MetadataProvider metadataProvider, IHyracksClientConnection hcc) throws Exception { + MetadataTransactionContext mdTxnCtx = MetadataManager.INSTANCE.beginTransaction(); + metadataProvider.setMetadataTxnContext(mdTxnCtx); + Dataset sourceDataset; + Dataset targetDataset; + // Generates the rebalance target files. While doing that, hold read locks on the dataset so + // that no one can drop the rebalance source dataset. + try { + // The source dataset. + sourceDataset = metadataProvider.findDataset(dataverseName, datasetName); + + // If the source dataset doesn't exist, then it's a no-op. + if (sourceDataset == null) { + return; + } + + Set sourceNodes = new HashSet<>(metadataProvider.findNodes(sourceDataset.getNodeGroupName())); + + // The the source nodes are identical to the target nodes. + if (sourceNodes.equals(targetNcNames)) { + return; + } + + // The new node group name. + String nodeGroupName = datasetName + "_" + (sourceDataset.getRebalanceCount() + 1); + + // Creates a node group for rebalance. + createNodeGroup(nodeGroupName, targetNcNames, metadataProvider); + + // The target dataset for rebalance. + targetDataset = new Dataset(sourceDataset, true, nodeGroupName); + + // Rebalances the source dataset into the target dataset. + rebalance(sourceDataset, targetDataset, metadataProvider, hcc); + + // Complete the metadata transaction. + MetadataManager.INSTANCE.commitTransaction(mdTxnCtx); + } catch (Exception e) { + abort(e, e, mdTxnCtx); + throw e; + } finally { + metadataProvider.getLocks().reset(); + } + + // Starts another transaction for switching the metadata entity. + mdTxnCtx = MetadataManager.INSTANCE.beginTransaction(); + metadataProvider.setMetadataTxnContext(mdTxnCtx); + try { + // Atomically switches the rebalance target to become the source dataset. + rebalanceSwitch(sourceDataset, targetDataset, metadataProvider, hcc); + + // Complete the metadata transaction. + MetadataManager.INSTANCE.commitTransaction(mdTxnCtx); + } catch (Exception e) { + abort(e, e, mdTxnCtx); + throw e; + } finally { + metadataProvider.getLocks().reset(); + } + } + + // Creates a node group for the rebalance target dataset. + private static void createNodeGroup(String ngName, Set ncNames, MetadataProvider metadataProvider) + throws Exception { + String nodeGroup = ngName; + MetadataTransactionContext mdTxnCtx = metadataProvider.getMetadataTxnContext(); + MetadataLockManager.INSTANCE.acquireNodeGroupWriteLock(metadataProvider.getLocks(), nodeGroup); + NodeGroup ng = MetadataManager.INSTANCE.getNodegroup(mdTxnCtx, nodeGroup); + if (ng != null) { + nodeGroup = ngName + UUID.randomUUID().toString(); + MetadataLockManager.INSTANCE.acquireNodeGroupWriteLock(metadataProvider.getLocks(), nodeGroup); + } + MetadataManager.INSTANCE.addNodegroup(mdTxnCtx, new NodeGroup(nodeGroup, new ArrayList<>(ncNames))); + } + + // Rebalances from the source to the target. + private static void rebalance(Dataset source, Dataset target, MetadataProvider metadataProvider, + IHyracksClientConnection hcc) throws Exception { + // Creates the rebalance target. + createRebalanceTarget(target, metadataProvider, hcc); + + // Populates the data from the rebalance source to the rebalance target. + populateDataToRebalanceTarget(source, target, metadataProvider, hcc); + } + + private static void rebalanceSwitch(Dataset source, Dataset target, MetadataProvider metadataProvider, + IHyracksClientConnection hcc) throws Exception { + MetadataTransactionContext mdTxnCtx = metadataProvider.getMetadataTxnContext(); + + // Acquires the metadata write lock for the source/target dataset. + writeLockDataset(metadataProvider.getLocks(), source); + + Dataset sourceDataset = MetadataManagerUtil.findDataset(mdTxnCtx, source.getDataverseName(), + source.getDatasetName()); + if (sourceDataset == null) { + // The dataset has already been dropped. + // In this case, we should drop the generated target dataset files. + dropDatasetFiles(target, metadataProvider, hcc); + return; + } + + // Drops the source dataset files. + dropDatasetFiles(source, metadataProvider, hcc); + + // Drops the source dataset's metadata entry. + MetadataManager.INSTANCE.dropDataset(mdTxnCtx, source.getDataverseName(), source.getDatasetName()); + + // Drops the metadata entry of source dataset's node group. + String sourceNodeGroup = source.getNodeGroupName(); + if (!sourceNodeGroup.equals(MetadataConstants.METADATA_DEFAULT_NODEGROUP_NAME)) { + MetadataManager.INSTANCE.dropNodegroup(mdTxnCtx, sourceNodeGroup); + } + + // Adds a new rebalanced dataset entry in the metadata storage + MetadataManager.INSTANCE.addDataset(mdTxnCtx, target); + } + + // Creates the files for the rebalance target dataset. + private static void createRebalanceTarget(Dataset target, MetadataProvider metadataProvider, + IHyracksClientConnection hcc) throws Exception { + JobSpecification spec = DatasetUtil.createDatasetJobSpec(target, metadataProvider); + JobUtils.runJob(hcc, spec, true); + } + + // Populates the data from the source dataset to the rebalance target dataset. + private static void populateDataToRebalanceTarget(Dataset source, Dataset target, MetadataProvider metadataProvider, + IHyracksClientConnection hcc) throws Exception { + JobSpecification spec = new JobSpecification(); + JobId jobId = JobIdFactory.generateJobId(); + JobEventListenerFactory jobEventListenerFactory = new JobEventListenerFactory(jobId, true); + spec.setJobletEventListenerFactory(jobEventListenerFactory); + + // The pipeline starter. + IOperatorDescriptor starter = DatasetUtil.createDummyKeyProviderOp(spec, source, metadataProvider); + + // Creates primary index scan op. + IOperatorDescriptor primaryScanOp = DatasetUtil.createPrimaryIndexScanOp(spec, metadataProvider, source, jobId); + + // Creates secondary BTree upsert op. + IOperatorDescriptor upsertOp = createPrimaryIndexUpsertOp(spec, metadataProvider, source, target); + + // The final commit operator. + IOperatorDescriptor commitOp = createUpsertCommitOp(spec, metadataProvider, jobId, target); + + // Connects empty-tuple-source and scan. + spec.connect(new OneToOneConnectorDescriptor(spec), starter, 0, primaryScanOp, 0); + + // Connects scan and upsert. + int numKeys = target.getPrimaryKeys().size(); + int[] keys = IntStream.range(0, numKeys).toArray(); + IConnectorDescriptor connectorDescriptor = new MToNPartitioningConnectorDescriptor(spec, + new FieldHashPartitionComputerFactory(keys, target.getPrimaryHashFunctionFactories(metadataProvider))); + spec.connect(connectorDescriptor, primaryScanOp, 0, upsertOp, 0); + + // Connects upsert and sink. + spec.connect(new OneToOneConnectorDescriptor(spec), upsertOp, 0, commitOp, 0); + + // Executes the job. + JobUtils.runJob(hcc, spec, true); + } + + // Creates the primary index upsert operator for populating the target dataset. + private static IOperatorDescriptor createPrimaryIndexUpsertOp(JobSpecification spec, + MetadataProvider metadataProvider, Dataset source, Dataset target) throws AlgebricksException { + int numKeys = source.getPrimaryKeys().size(); + int numValues = source.hasMetaPart() ? 2 : 1; + int[] fieldPermutation = IntStream.range(0, numKeys + numValues).toArray(); + Pair upsertOpAndConstraints = DatasetUtil + .createPrimaryIndexUpsertOp(spec, metadataProvider, target, + source.getPrimaryRecordDescriptor(metadataProvider), fieldPermutation, + MissingWriterFactory.INSTANCE); + IOperatorDescriptor upsertOp = upsertOpAndConstraints.first; + AlgebricksPartitionConstraintHelper.setPartitionConstraintInJobSpec(spec, upsertOp, + upsertOpAndConstraints.second); + return upsertOp; + } + + // Creates the commit operator for populating the target dataset. + private static IOperatorDescriptor createUpsertCommitOp(JobSpecification spec, MetadataProvider metadataProvider, + JobId jobId, Dataset target) throws AlgebricksException { + int numKeys = target.getPrimaryKeys().size(); + int[] primaryKeyFields = IntStream.range(0, numKeys).toArray(); + return new AlgebricksMetaOperatorDescriptor(spec, 1, 0, + new IPushRuntimeFactory[] { + target.getCommitRuntimeFactory(metadataProvider, jobId, primaryKeyFields, true) }, + new RecordDescriptor[] { target.getPrimaryRecordDescriptor(metadataProvider) }); + } + + private static void dropDatasetFiles(Dataset dataset, MetadataProvider metadataProvider, + IHyracksClientConnection hcc) throws Exception { + List jobs = new ArrayList<>(); + List indexes = metadataProvider.getDatasetIndexes(dataset.getDataverseName(), dataset.getDatasetName()); + for (Index index : indexes) { + jobs.add(IndexUtil.buildDropIndexJobSpec(index, metadataProvider, dataset)); + } + for (JobSpecification jobSpec : jobs) { + JobUtils.runJob(hcc, jobSpec, true); + } + } + + // Acquires a read lock for the dataverse and a write lock for the dataset, in order to populate the dataset. + private static void writeLockDataset(LockList locks, Dataset dataset) throws AsterixException { + MetadataLockManager.INSTANCE.acquireDataverseReadLock(locks, dataset.getDataverseName()); + MetadataLockManager.INSTANCE.acquireDatasetWriteLock(locks, + dataset.getDataverseName() + "." + dataset.getDatasetName()); + } +} http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java index 856638f..cbe551b 100644 --- a/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java +++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/app/bootstrap/TestNodeController.java @@ -406,7 +406,7 @@ public class TestNodeController { IndexType.BTREE, keyFieldNames, keyFieldSourceIndicators, keyFieldTypes, false, true, MetadataUtil.PENDING_NO_OP); List nodes = Collections.singletonList(ExecutionTestUtil.integrationUtil.ncs[0].getId()); - FileSplit[] splits = SplitsAndConstraintsUtil.getDatasetSplits(dataset, nodes, index.getIndexName(), false); + FileSplit[] splits = SplitsAndConstraintsUtil.getIndexSplits(dataset, index.getIndexName(), nodes); fileSplitProvider = new ConstantFileSplitProvider(Arrays.copyOfRange(splits, 0, 1)); } http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/RebalanceTest.java ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/RebalanceTest.java b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/RebalanceTest.java new file mode 100644 index 0000000..7eb1110 --- /dev/null +++ b/asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/RebalanceTest.java @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.asterix.test.runtime; + +import java.util.Collection; + +import org.apache.asterix.test.common.TestExecutor; +import org.apache.asterix.testframework.context.TestCaseContext; +import org.junit.AfterClass; +import org.junit.BeforeClass; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.junit.runners.Parameterized; +import org.junit.runners.Parameterized.Parameters; + +/** + * Runs rebalance tests. + */ +@RunWith(Parameterized.class) +public class RebalanceTest { + protected static final String TEST_CONFIG_FILE_NAME = "asterix-build-configuration.xml"; + + @BeforeClass + public static void setUp() throws Exception { + LangExecutionUtil.setUp(TEST_CONFIG_FILE_NAME, new TestExecutor()); + } + + @AfterClass + public static void tearDown() throws Exception { + LangExecutionUtil.tearDown(); + } + + @Parameters(name = "RebalanceTest {index}: {0}") + public static Collection tests() throws Exception { + return LangExecutionUtil.tests("only_sqlpp.xml", "rebalance.xml"); + } + + protected TestCaseContext tcCtx; + + public RebalanceTest(TestCaseContext tcCtx) { + this.tcCtx = tcCtx; + } + + @Test + public void test() throws Exception { + LangExecutionUtil.test(tcCtx); + } +} http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.1.ddl.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.1.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.1.ddl.sqlpp new file mode 100644 index 0000000..92698ab --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.1.ddl.sqlpp @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +drop dataverse tpch if exists; +create dataverse tpch; + +use tpch; + + +create type tpch.LineItemType as + closed { + l_orderkey : bigint, + l_partkey : bigint, + l_suppkey : bigint, + l_linenumber : bigint, + l_quantity : double, + l_extendedprice : double, + l_discount : double, + l_tax : double, + l_returnflag : string, + l_linestatus : string, + l_shipdate : string, + l_commitdate : string, + l_receiptdate : string, + l_shipinstruct : string, + l_shipmode : string, + l_comment : string +} + +create dataset LineItem(LineItemType) primary key l_orderkey,l_linenumber; + http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.2.update.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.2.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.2.update.sqlpp new file mode 100644 index 0000000..8a59946 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.2.update.sqlpp @@ -0,0 +1,23 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +use tpch; + +load dataset LineItem using localfs ((`path`=`asterix_nc1://data/tpch0.001/lineitem.tbl`), + (`format`=`delimited-text`),(`delimiter`=`|`)); http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.3.post.http ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.3.post.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.3.post.http new file mode 100644 index 0000000..f5b4573 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.3.post.http @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/admin/rebalance?dataverseName=tpch&datasetName=LineItem&nodes="asterix_nc1,asterix_nc1,asterix_nc1" http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.4.query.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.4.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.4.query.sqlpp new file mode 100644 index 0000000..9b19e48 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.4.query.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +select DatasetName, GroupName, rebalanceCount +from Metadata.`Dataset` +where DatasetName = "LineItem"; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.5.query.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.5.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.5.query.sqlpp new file mode 100644 index 0000000..6045a79 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/duplicate_location/duplicate_location.5.query.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +select GroupName, NodeNames +from Metadata.`Nodegroup` +where GroupName like "LineItem%"; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.1.ddl.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.1.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.1.ddl.sqlpp new file mode 100644 index 0000000..92698ab --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.1.ddl.sqlpp @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +drop dataverse tpch if exists; +create dataverse tpch; + +use tpch; + + +create type tpch.LineItemType as + closed { + l_orderkey : bigint, + l_partkey : bigint, + l_suppkey : bigint, + l_linenumber : bigint, + l_quantity : double, + l_extendedprice : double, + l_discount : double, + l_tax : double, + l_returnflag : string, + l_linestatus : string, + l_shipdate : string, + l_commitdate : string, + l_receiptdate : string, + l_shipinstruct : string, + l_shipmode : string, + l_comment : string +} + +create dataset LineItem(LineItemType) primary key l_orderkey,l_linenumber; + http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.2.update.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.2.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.2.update.sqlpp new file mode 100644 index 0000000..8a59946 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.2.update.sqlpp @@ -0,0 +1,23 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +use tpch; + +load dataset LineItem using localfs ((`path`=`asterix_nc1://data/tpch0.001/lineitem.tbl`), + (`format`=`delimited-text`),(`delimiter`=`|`)); http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.3.post.http ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.3.post.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.3.post.http new file mode 100644 index 0000000..e9e0e34 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.3.post.http @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/admin/rebalance?dataverseName=tpch&datasetName=LineItem&nodes="" http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.4.query.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.4.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.4.query.sqlpp new file mode 100644 index 0000000..9b19e48 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/empty_location/empty_location.4.query.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +select DatasetName, GroupName, rebalanceCount +from Metadata.`Dataset` +where DatasetName = "LineItem"; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.1.ddl.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.1.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.1.ddl.sqlpp new file mode 100644 index 0000000..92698ab --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.1.ddl.sqlpp @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +drop dataverse tpch if exists; +create dataverse tpch; + +use tpch; + + +create type tpch.LineItemType as + closed { + l_orderkey : bigint, + l_partkey : bigint, + l_suppkey : bigint, + l_linenumber : bigint, + l_quantity : double, + l_extendedprice : double, + l_discount : double, + l_tax : double, + l_returnflag : string, + l_linestatus : string, + l_shipdate : string, + l_commitdate : string, + l_receiptdate : string, + l_shipinstruct : string, + l_shipmode : string, + l_comment : string +} + +create dataset LineItem(LineItemType) primary key l_orderkey,l_linenumber; + http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.2.update.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.2.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.2.update.sqlpp new file mode 100644 index 0000000..8a59946 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.2.update.sqlpp @@ -0,0 +1,23 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +use tpch; + +load dataset LineItem using localfs ((`path`=`asterix_nc1://data/tpch0.001/lineitem.tbl`), + (`format`=`delimited-text`),(`delimiter`=`|`)); http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.3.post.http ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.3.post.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.3.post.http new file mode 100644 index 0000000..33203f1 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.3.post.http @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/admin/rebalance?dataverseName=tpch&datasetName=LineItem&nodes="asterix_nc2,asterix_nc1" http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.4.query.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.4.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.4.query.sqlpp new file mode 100644 index 0000000..9b19e48 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/identical_location/identical_location.4.query.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +select DatasetName, GroupName, rebalanceCount +from Metadata.`Dataset` +where DatasetName = "LineItem"; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/nonexist_dataset/nonexist_dataset.1.post.http ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/nonexist_dataset/nonexist_dataset.1.post.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/nonexist_dataset/nonexist_dataset.1.post.http new file mode 100644 index 0000000..737c1fe --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/nonexist_dataset/nonexist_dataset.1.post.http @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/admin/rebalance?dataverseName=tpch&datasetName=LineItem&nodes="asterix_nc1" http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.1.ddl.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.1.ddl.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.1.ddl.sqlpp new file mode 100644 index 0000000..92698ab --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.1.ddl.sqlpp @@ -0,0 +1,47 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +drop dataverse tpch if exists; +create dataverse tpch; + +use tpch; + + +create type tpch.LineItemType as + closed { + l_orderkey : bigint, + l_partkey : bigint, + l_suppkey : bigint, + l_linenumber : bigint, + l_quantity : double, + l_extendedprice : double, + l_discount : double, + l_tax : double, + l_returnflag : string, + l_linestatus : string, + l_shipdate : string, + l_commitdate : string, + l_receiptdate : string, + l_shipinstruct : string, + l_shipmode : string, + l_comment : string +} + +create dataset LineItem(LineItemType) primary key l_orderkey,l_linenumber; + http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.10.query.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.10.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.10.query.sqlpp new file mode 100644 index 0000000..ca42f3c --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.10.query.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +use tpch; + +select value count(*) from LineItem; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.11.query.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.11.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.11.query.sqlpp new file mode 100644 index 0000000..9b19e48 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.11.query.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +select DatasetName, GroupName, rebalanceCount +from Metadata.`Dataset` +where DatasetName = "LineItem"; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.2.update.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.2.update.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.2.update.sqlpp new file mode 100644 index 0000000..8a59946 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.2.update.sqlpp @@ -0,0 +1,23 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +use tpch; + +load dataset LineItem using localfs ((`path`=`asterix_nc1://data/tpch0.001/lineitem.tbl`), + (`format`=`delimited-text`),(`delimiter`=`|`)); http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.3.query.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.3.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.3.query.sqlpp new file mode 100644 index 0000000..ca42f3c --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.3.query.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +use tpch; + +select value count(*) from LineItem; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.4.post.http ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.4.post.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.4.post.http new file mode 100644 index 0000000..737c1fe --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.4.post.http @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/admin/rebalance?dataverseName=tpch&datasetName=LineItem&nodes="asterix_nc1" http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.5.get.http ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.5.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.5.get.http new file mode 100644 index 0000000..360a01f --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.5.get.http @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/connector?dataverseName=tpch&datasetName=LineItem http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.6.query.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.6.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.6.query.sqlpp new file mode 100644 index 0000000..ca42f3c --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.6.query.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +use tpch; + +select value count(*) from LineItem; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.7.query.sqlpp ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.7.query.sqlpp b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.7.query.sqlpp new file mode 100644 index 0000000..9b19e48 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.7.query.sqlpp @@ -0,0 +1,22 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +select DatasetName, GroupName, rebalanceCount +from Metadata.`Dataset` +where DatasetName = "LineItem"; http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.8.post.http ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.8.post.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.8.post.http new file mode 100644 index 0000000..1a677fc --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.8.post.http @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/admin/rebalance?dataverseName=tpch&datasetName=LineItem&nodes="asterix_nc1,asterix_nc2" http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.9.get.http ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.9.get.http b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.9.get.http new file mode 100644 index 0000000..360a01f --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/rebalance/single_dataset/single_dataset.9.get.http @@ -0,0 +1,20 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ + +/connector?dataverseName=tpch&datasetName=LineItem http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/rebalance.xml ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/rebalance.xml b/asterixdb/asterix-app/src/test/resources/runtimets/rebalance.xml new file mode 100644 index 0000000..aa51361 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/rebalance.xml @@ -0,0 +1,50 @@ + + + + + + duplicate_location + + + + + empty_location + HTTP operation failed: +STATUS LINE: HTTP/1.1 400 Bad Request +ERROR_BODY: {"results":"target nodes should not be empty"} + + + + + identical_location + + + + + nonexist_dataset + + + + + single_dataset + + + + http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.3.adm ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.3.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.3.adm new file mode 100644 index 0000000..4b9eb7d --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.3.adm @@ -0,0 +1 @@ +{"results":"successful"} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.4.adm ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.4.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.4.adm new file mode 100644 index 0000000..2760db0 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.4.adm @@ -0,0 +1 @@ +{ "DatasetName": "LineItem", "GroupName": "LineItem_1", "rebalanceCount": 1 } \ No newline at end of file http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.5.adm ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.5.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.5.adm new file mode 100644 index 0000000..5ec3f11 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/duplicate_location/duplicate_location.5.adm @@ -0,0 +1 @@ +{ "GroupName": "LineItem_1", "NodeNames": {{ "asterix_nc1" }} } \ No newline at end of file http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/empty_location/empty_location.3.adm ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/empty_location/empty_location.3.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/empty_location/empty_location.3.adm new file mode 100644 index 0000000..398b082 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/empty_location/empty_location.3.adm @@ -0,0 +1 @@ +{ "DatasetName": "LineItem", "GroupName": "DEFAULT_NG_ALL_NODES" } \ No newline at end of file http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/identical_location/identical_location.3.adm ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/identical_location/identical_location.3.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/identical_location/identical_location.3.adm new file mode 100644 index 0000000..4b9eb7d --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/identical_location/identical_location.3.adm @@ -0,0 +1 @@ +{"results":"successful"} \ No newline at end of file http://git-wip-us.apache.org/repos/asf/asterixdb/blob/87805197/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/identical_location/identical_location.4.adm ---------------------------------------------------------------------- diff --git a/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/identical_location/identical_location.4.adm b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/identical_location/identical_location.4.adm new file mode 100644 index 0000000..398b082 --- /dev/null +++ b/asterixdb/asterix-app/src/test/resources/runtimets/results/rebalance/identical_location/identical_location.4.adm @@ -0,0 +1 @@ +{ "DatasetName": "LineItem", "GroupName": "DEFAULT_NG_ALL_NODES" } \ No newline at end of file