Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 562AA200C06 for ; Fri, 13 Jan 2017 02:21:06 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 53272160B4C; Fri, 13 Jan 2017 01:21:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CB5B2160B40 for ; Fri, 13 Jan 2017 02:21:04 +0100 (CET) Received: (qmail 75383 invoked by uid 500); 13 Jan 2017 01:21:03 -0000 Mailing-List: contact commits-help@quickstep.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@quickstep.incubator.apache.org Delivered-To: mailing list commits@quickstep.incubator.apache.org Received: (qmail 75374 invoked by uid 99); 13 Jan 2017 01:21:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jan 2017 01:21:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 5DA4E1A0121 for ; Fri, 13 Jan 2017 01:21:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.219 X-Spam-Level: X-Spam-Status: No, score=-6.219 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id hAFej2t1Oddk for ; Fri, 13 Jan 2017 01:20:57 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with SMTP id B3F005F370 for ; Fri, 13 Jan 2017 01:20:56 +0000 (UTC) Received: (qmail 75056 invoked by uid 99); 13 Jan 2017 01:20:56 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jan 2017 01:20:56 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 2B225DFA22; Fri, 13 Jan 2017 01:20:56 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: zuyuz@apache.org To: commits@quickstep.incubator.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: incubator-quickstep git commit: Added optimizer support regarding hash partitions. [Forced Update!] Date: Fri, 13 Jan 2017 01:20:56 +0000 (UTC) archived-at: Fri, 13 Jan 2017 01:21:06 -0000 Repository: incubator-quickstep Updated Branches: refs/heads/quickstep_partition_parser_support 33680b92c -> 6b063893c (forced update) Added optimizer support regarding hash partitions. - CreateTable - Insert / InsertSelection - UpdateTable Project: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/commit/6b063893 Tree: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/tree/6b063893 Diff: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/diff/6b063893 Branch: refs/heads/quickstep_partition_parser_support Commit: 6b063893cac0703e1c13fa6c7633e15c8d1db183 Parents: 2229423 Author: Zuyu Zhang Authored: Tue Jan 10 17:00:08 2017 -0800 Committer: Zuyu Zhang Committed: Thu Jan 12 17:09:03 2017 -0800 ---------------------------------------------------------------------- query_optimizer/CMakeLists.txt | 2 + query_optimizer/ExecutionGenerator.cpp | 46 ++++++-- query_optimizer/logical/CMakeLists.txt | 1 + query_optimizer/logical/CreateTable.hpp | 22 +++- query_optimizer/physical/CMakeLists.txt | 1 + query_optimizer/physical/CreateTable.hpp | 22 +++- query_optimizer/resolver/CMakeLists.txt | 3 + query_optimizer/resolver/Resolver.cpp | 106 ++++++++++++++----- query_optimizer/resolver/Resolver.hpp | 13 ++- query_optimizer/strategy/OneToOne.cpp | 3 +- .../tests/execution_generator/CMakeLists.txt | 6 ++ .../tests/execution_generator/Create.test | 9 ++ .../tests/execution_generator/Partition.test | 54 ++++++++++ .../tests/logical_generator/Create.test | 10 ++ .../tests/physical_generator/Create.test | 18 ++++ query_optimizer/tests/resolver/Create.test | 10 ++ 16 files changed, 278 insertions(+), 48 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/CMakeLists.txt ---------------------------------------------------------------------- diff --git a/query_optimizer/CMakeLists.txt b/query_optimizer/CMakeLists.txt index 10c52a1..1a15271 100644 --- a/query_optimizer/CMakeLists.txt +++ b/query_optimizer/CMakeLists.txt @@ -63,6 +63,8 @@ target_link_libraries(quickstep_queryoptimizer_ExecutionGenerator quickstep_catalog_CatalogRelation quickstep_catalog_CatalogRelationSchema quickstep_catalog_CatalogTypedefs + quickstep_catalog_PartitionScheme + quickstep_catalog_PartitionSchemeHeader quickstep_expressions_Expressions_proto quickstep_expressions_aggregation_AggregateFunction quickstep_expressions_aggregation_AggregateFunction_proto http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/ExecutionGenerator.cpp ---------------------------------------------------------------------- diff --git a/query_optimizer/ExecutionGenerator.cpp b/query_optimizer/ExecutionGenerator.cpp index 29e67f7..2133722 100644 --- a/query_optimizer/ExecutionGenerator.cpp +++ b/query_optimizer/ExecutionGenerator.cpp @@ -44,6 +44,8 @@ #include "catalog/CatalogRelation.hpp" #include "catalog/CatalogRelationSchema.hpp" #include "catalog/CatalogTypedefs.hpp" +#include "catalog/PartitionScheme.hpp" +#include "catalog/PartitionSchemeHeader.hpp" #include "expressions/Expressions.pb.h" #include "expressions/aggregation/AggregateFunction.hpp" #include "expressions/aggregation/AggregateFunction.pb.h" @@ -1017,6 +1019,11 @@ void ExecutionGenerator::convertCreateTable( catalog_relation->setDefaultStorageBlockLayout(layout.release()); } + if (physical_plan->partition_scheme_header_proto()) { + catalog_relation->setPartitionScheme(new PartitionScheme( + PartitionSchemeHeader::ReconstructFromProto(*physical_plan->partition_scheme_header_proto()))); + } + execution_plan_->addRelationalOperator( new CreateTableOperator(query_handle_->query_id(), catalog_relation.release(), @@ -1139,14 +1146,21 @@ void ExecutionGenerator::convertInsertTuple( query_context_proto_->insert_destinations_size(); S::InsertDestination *insert_destination_proto = query_context_proto_->add_insert_destinations(); - insert_destination_proto->set_insert_destination_type(S::InsertDestinationType::BLOCK_POOL); insert_destination_proto->set_relation_id(input_relation.getID()); insert_destination_proto->mutable_layout()->MergeFrom( input_relation.getDefaultStorageBlockLayout().getDescription()); - const vector blocks(input_relation.getBlocksSnapshot()); - for (const block_id block : blocks) { - insert_destination_proto->AddExtension(S::BlockPoolInsertDestination::blocks, block); + if (input_relation.hasPartitionScheme()) { + insert_destination_proto->set_insert_destination_type(S::InsertDestinationType::PARTITION_AWARE); + insert_destination_proto->MutableExtension(S::PartitionAwareInsertDestination::partition_scheme) + ->MergeFrom(input_relation.getPartitionScheme().getProto()); + } else { + insert_destination_proto->set_insert_destination_type(S::InsertDestinationType::BLOCK_POOL); + + const vector blocks(input_relation.getBlocksSnapshot()); + for (const block_id block : blocks) { + insert_destination_proto->AddExtension(S::BlockPoolInsertDestination::blocks, block); + } } const QueryPlan::DAGNodeIndex insert_operator_index = @@ -1197,15 +1211,21 @@ void ExecutionGenerator::convertInsertSelection( const QueryContext::insert_destination_id insert_destination_index = query_context_proto_->insert_destinations_size(); S::InsertDestination *insert_destination_proto = query_context_proto_->add_insert_destinations(); - - insert_destination_proto->set_insert_destination_type(S::InsertDestinationType::BLOCK_POOL); insert_destination_proto->set_relation_id(destination_relation.getID()); insert_destination_proto->mutable_layout()->MergeFrom( destination_relation.getDefaultStorageBlockLayout().getDescription()); - const vector blocks(destination_relation.getBlocksSnapshot()); - for (const block_id block : blocks) { - insert_destination_proto->AddExtension(S::BlockPoolInsertDestination::blocks, block); + if (destination_relation.hasPartitionScheme()) { + insert_destination_proto->set_insert_destination_type(S::InsertDestinationType::PARTITION_AWARE); + insert_destination_proto->MutableExtension(S::PartitionAwareInsertDestination::partition_scheme) + ->MergeFrom(destination_relation.getPartitionScheme().getProto()); + } else { + insert_destination_proto->set_insert_destination_type(S::InsertDestinationType::BLOCK_POOL); + + const vector blocks(destination_relation.getBlocksSnapshot()); + for (const block_id block : blocks) { + insert_destination_proto->AddExtension(S::BlockPoolInsertDestination::blocks, block); + } } const CatalogRelationInfo *selection_relation_info = @@ -1274,7 +1294,13 @@ void ExecutionGenerator::convertUpdateTable( query_context_proto_->insert_destinations_size(); S::InsertDestination *relocation_destination_proto = query_context_proto_->add_insert_destinations(); - relocation_destination_proto->set_insert_destination_type(S::InsertDestinationType::BLOCK_POOL); + if (input_relation->hasPartitionScheme()) { + relocation_destination_proto->set_insert_destination_type(S::InsertDestinationType::PARTITION_AWARE); + relocation_destination_proto->MutableExtension(S::PartitionAwareInsertDestination::partition_scheme) + ->MergeFrom(input_relation->getPartitionScheme().getProto()); + } else { + relocation_destination_proto->set_insert_destination_type(S::InsertDestinationType::BLOCK_POOL); + } relocation_destination_proto->set_relation_id(input_rel_id); // Convert the predicate proto. http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/logical/CMakeLists.txt ---------------------------------------------------------------------- diff --git a/query_optimizer/logical/CMakeLists.txt b/query_optimizer/logical/CMakeLists.txt index c67f96f..8aca550 100644 --- a/query_optimizer/logical/CMakeLists.txt +++ b/query_optimizer/logical/CMakeLists.txt @@ -86,6 +86,7 @@ target_link_libraries(quickstep_queryoptimizer_logical_CreateIndex quickstep_utility_Macros) target_link_libraries(quickstep_queryoptimizer_logical_CreateTable glog + quickstep_catalog_Catalog_proto quickstep_queryoptimizer_OptimizerTree quickstep_queryoptimizer_expressions_AttributeReference quickstep_queryoptimizer_logical_Logical http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/logical/CreateTable.hpp ---------------------------------------------------------------------- diff --git a/query_optimizer/logical/CreateTable.hpp b/query_optimizer/logical/CreateTable.hpp index cc7c6d7..d4f16b6 100644 --- a/query_optimizer/logical/CreateTable.hpp +++ b/query_optimizer/logical/CreateTable.hpp @@ -24,6 +24,7 @@ #include #include +#include "catalog/Catalog.pb.h" #include "query_optimizer/OptimizerTree.hpp" #include "query_optimizer/expressions/AttributeReference.hpp" #include "query_optimizer/logical/Logical.hpp" @@ -72,10 +73,17 @@ class CreateTable : public Logical { return block_properties_; } + /** + * @return Shared pointer to the serialized partition scheme header. + */ + const std::shared_ptr partition_scheme_header_proto() const { + return partition_scheme_header_proto_; + } + LogicalPtr copyWithNewChildren( const std::vector &new_children) const override { DCHECK_EQ(getNumChildren(), new_children.size()); - return Create(relation_name_, attributes_, block_properties_); + return Create(relation_name_, attributes_, block_properties_, partition_scheme_header_proto_); } std::vector getOutputAttributes() const override { @@ -100,8 +108,9 @@ class CreateTable : public Logical { static CreateTablePtr Create( const std::string &relation_name, const std::vector &attributes, - const std::shared_ptr &block_properties) { - return CreateTablePtr(new CreateTable(relation_name, attributes, block_properties)); + const std::shared_ptr &block_properties, + const std::shared_ptr &partition_scheme_header_proto) { + return CreateTablePtr(new CreateTable(relation_name, attributes, block_properties, partition_scheme_header_proto)); } protected: @@ -117,17 +126,20 @@ class CreateTable : public Logical { CreateTable( const std::string &relation_name, const std::vector &attributes, - const std::shared_ptr &block_properties) + const std::shared_ptr &block_properties, + const std::shared_ptr &partition_scheme_header_proto) : relation_name_(relation_name), attributes_(attributes), block_properties_(block_properties), block_properties_representation_( - getOptimizerRepresentationForProto(block_properties_.get())) {} + getOptimizerRepresentationForProto(block_properties_.get())), + partition_scheme_header_proto_(partition_scheme_header_proto) {} std::string relation_name_; std::vector attributes_; std::shared_ptr block_properties_; std::shared_ptr > block_properties_representation_; + std::shared_ptr partition_scheme_header_proto_; DISALLOW_COPY_AND_ASSIGN(CreateTable); }; http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/physical/CMakeLists.txt ---------------------------------------------------------------------- diff --git a/query_optimizer/physical/CMakeLists.txt b/query_optimizer/physical/CMakeLists.txt index 5c2cd0b..7f26943 100644 --- a/query_optimizer/physical/CMakeLists.txt +++ b/query_optimizer/physical/CMakeLists.txt @@ -85,6 +85,7 @@ target_link_libraries(quickstep_queryoptimizer_physical_CreateIndex quickstep_utility_Macros) target_link_libraries(quickstep_queryoptimizer_physical_CreateTable glog + quickstep_catalog_Catalog_proto quickstep_queryoptimizer_OptimizerTree quickstep_queryoptimizer_expressions_AttributeReference quickstep_queryoptimizer_expressions_ExpressionUtil http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/physical/CreateTable.hpp ---------------------------------------------------------------------- diff --git a/query_optimizer/physical/CreateTable.hpp b/query_optimizer/physical/CreateTable.hpp index 8e3bbd4..f982e32 100644 --- a/query_optimizer/physical/CreateTable.hpp +++ b/query_optimizer/physical/CreateTable.hpp @@ -24,6 +24,7 @@ #include #include +#include "catalog/Catalog.pb.h" #include "query_optimizer/OptimizerTree.hpp" #include "query_optimizer/expressions/AttributeReference.hpp" #include "query_optimizer/expressions/ExpressionUtil.hpp" @@ -75,10 +76,17 @@ class CreateTable : public Physical { return block_properties_; } + /** + * @return Shared pointer to the serialized partition scheme header. + */ + const std::shared_ptr partition_scheme_header_proto() const { + return partition_scheme_header_proto_; + } + PhysicalPtr copyWithNewChildren( const std::vector &new_children) const override { DCHECK_EQ(getNumChildren(), new_children.size()); - return Create(relation_name_, attributes_, block_properties_); + return Create(relation_name_, attributes_, block_properties_, partition_scheme_header_proto_); } std::vector getOutputAttributes() const override { @@ -107,8 +115,9 @@ class CreateTable : public Physical { static CreateTablePtr Create( const std::string &relation_name, const std::vector &attributes, - const std::shared_ptr &block_properties) { - return CreateTablePtr(new CreateTable(relation_name, attributes, block_properties)); + const std::shared_ptr &block_properties, + const std::shared_ptr &partition_scheme_header_proto) { + return CreateTablePtr(new CreateTable(relation_name, attributes, block_properties, partition_scheme_header_proto)); } protected: @@ -124,17 +133,20 @@ class CreateTable : public Physical { CreateTable( const std::string &relation_name, const std::vector &attributes, - const std::shared_ptr &block_properties) + const std::shared_ptr &block_properties, + const std::shared_ptr &partition_scheme_header_proto) : relation_name_(relation_name), attributes_(attributes), block_properties_(block_properties), block_properties_representation_( - getOptimizerRepresentationForProto(block_properties_.get())) {} + getOptimizerRepresentationForProto(block_properties_.get())), + partition_scheme_header_proto_(partition_scheme_header_proto) {} std::string relation_name_; std::vector attributes_; std::shared_ptr block_properties_; std::shared_ptr > block_properties_representation_; + std::shared_ptr partition_scheme_header_proto_; DISALLOW_COPY_AND_ASSIGN(CreateTable); }; http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/resolver/CMakeLists.txt ---------------------------------------------------------------------- diff --git a/query_optimizer/resolver/CMakeLists.txt b/query_optimizer/resolver/CMakeLists.txt index 5251ccc..a34273e 100644 --- a/query_optimizer/resolver/CMakeLists.txt +++ b/query_optimizer/resolver/CMakeLists.txt @@ -34,6 +34,8 @@ target_link_libraries(quickstep_queryoptimizer_resolver_NameResolver target_link_libraries(quickstep_queryoptimizer_resolver_Resolver glog quickstep_catalog_CatalogDatabase + quickstep_catalog_CatalogTypedefs + quickstep_catalog_Catalog_proto quickstep_expressions_aggregation_AggregateFunction quickstep_expressions_aggregation_AggregateFunctionFactory quickstep_expressions_tablegenerator_GeneratorFunction @@ -53,6 +55,7 @@ target_link_libraries(quickstep_queryoptimizer_resolver_Resolver quickstep_parser_ParseLimit quickstep_parser_ParseLiteralValue quickstep_parser_ParseOrderBy + quickstep_parser_ParsePartitionClause quickstep_parser_ParsePredicate quickstep_parser_ParsePredicateExists quickstep_parser_ParsePredicateInTableQuery http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/resolver/Resolver.cpp ---------------------------------------------------------------------- diff --git a/query_optimizer/resolver/Resolver.cpp b/query_optimizer/resolver/Resolver.cpp index 2580342..3c9faee 100644 --- a/query_optimizer/resolver/Resolver.cpp +++ b/query_optimizer/resolver/Resolver.cpp @@ -29,7 +29,9 @@ #include #include +#include "catalog/Catalog.pb.h" #include "catalog/CatalogDatabase.hpp" +#include "catalog/CatalogTypedefs.hpp" #include "expressions/aggregation/AggregateFunction.hpp" #include "expressions/aggregation/AggregateFunctionFactory.hpp" #include "expressions/table_generator/GeneratorFunction.hpp" @@ -49,6 +51,7 @@ #include "parser/ParseLimit.hpp" #include "parser/ParseLiteralValue.hpp" #include "parser/ParseOrderBy.hpp" +#include "parser/ParsePartitionClause.hpp" #include "parser/ParsePredicate.hpp" #include "parser/ParsePredicateExists.hpp" #include "parser/ParsePredicateInTableQuery.hpp" @@ -129,12 +132,15 @@ #include "glog/logging.h" +using std::make_unique; + namespace quickstep { namespace optimizer { namespace resolver { namespace E = ::quickstep::optimizer::expressions; namespace L = ::quickstep::optimizer::logical; +namespace S = ::quickstep::serialization; struct Resolver::ExpressionResolutionInfo { /** @@ -478,9 +484,32 @@ L::LogicalPtr Resolver::resolveCreateTable( std::shared_ptr block_properties(resolveBlockProperties(create_table_statement)); - return L::CreateTable::Create(relation_name, attributes, block_properties); + std::shared_ptr partition_scheme_header_proto( + resolvePartitionClause(create_table_statement)); + + return L::CreateTable::Create(relation_name, attributes, block_properties, partition_scheme_header_proto); +} + +namespace { + +attribute_id GetAttributeIdFromName(const PtrList&attribute_definition_list, + const std::string &attribute_name) { + const std::string lower_attribute_name = ToLower(attribute_name); + + attribute_id attr_id = 0; + for (const ParseAttributeDefinition &attribute_definition : attribute_definition_list) { + if (lower_attribute_name == ToLower(attribute_definition.name()->value())) { + return attr_id; + } + + ++attr_id; + } + + return kInvalidAttributeID; } +} // namespace + StorageBlockLayoutDescription* Resolver::resolveBlockProperties( const ParseStatementCreateTable &create_table_statement) { const ParseBlockProperties *block_properties @@ -540,25 +569,6 @@ StorageBlockLayoutDescription* Resolver::resolveBlockProperties( THROW_SQL_ERROR_AT(type_parse_string) << "Unrecognized storage type."; } - // Helper lambda function which will be used in COMPRESS and SORT resolution. - // Returns the column id from the name of the given attribute. Returns -1 if - // the attribute is not found. - auto columnIdFromAttributeName = [&create_table_statement]( - const std::string& attribute_name) -> int { - const std::string search_name = ToLower(attribute_name); - int i = 0; - for (const ParseAttributeDefinition &attribute_definition : - create_table_statement.attribute_definition_list()) { - const std::string lower_attribute_name = - ToLower(attribute_definition.name()->value()); - if (lower_attribute_name.compare(search_name) == 0) { - return i; - } - i++; - } - return -1; - }; - // Resolve the SORT property. const ParseString *sort_parse_string = block_properties->getSort(); if (block_requires_sort) { @@ -566,10 +576,10 @@ StorageBlockLayoutDescription* Resolver::resolveBlockProperties( THROW_SQL_ERROR_AT(type_parse_string) << "The SORT property must be specified as an attribute name."; } else { - const std::string &sort_name = sort_parse_string->value(); // Lookup the name and map to a column id. - int sort_id = columnIdFromAttributeName(sort_name); - if (sort_id == -1) { + const attribute_id sort_id = GetAttributeIdFromName(create_table_statement.attribute_definition_list(), + sort_parse_string->value()); + if (sort_id == kInvalidAttributeID) { THROW_SQL_ERROR_AT(sort_parse_string) << "The SORT property did not match any attribute name."; } else { @@ -609,8 +619,9 @@ StorageBlockLayoutDescription* Resolver::resolveBlockProperties( << "The COMPRESS property must be specified as ALL or a list of attributes."; } for (const ParseString &compressed_attribute_name : *compress_parse_strings) { - int column_id = columnIdFromAttributeName(compressed_attribute_name.value()); - if (column_id == -1) { + const attribute_id column_id = GetAttributeIdFromName(create_table_statement.attribute_definition_list(), + compressed_attribute_name.value()); + if (column_id == kInvalidAttributeID) { THROW_SQL_ERROR_AT(&compressed_attribute_name) << "The given attribute was not found."; } else { @@ -671,6 +682,51 @@ StorageBlockLayoutDescription* Resolver::resolveBlockProperties( return storage_block_description.release(); } +const S::PartitionSchemeHeader* Resolver::resolvePartitionClause( + const ParseStatementCreateTable &create_table_statement) { + const ParsePartitionClause *partition_clause = create_table_statement.opt_partition_clause(); + if (partition_clause == nullptr) { + return nullptr; + } + + const ParseString *partition_type_string = partition_clause->partition_type(); + if (partition_type_string == nullptr) { + THROW_SQL_ERROR_AT(partition_clause) + << "Partition type must be specified and be a string."; + } + + const PtrList &attribute_name_list = partition_clause->attribute_name_list(); + if (attribute_name_list.size() != 1) { + THROW_SQL_ERROR_AT(partition_clause) + << "Partition is supported on only one attribute."; + } + + const ParseString &partition_attribute_name = *(attribute_name_list.begin()); + const attribute_id attr_id = GetAttributeIdFromName(create_table_statement.attribute_definition_list(), + partition_attribute_name.value()); + if (attr_id == kInvalidAttributeID) { + THROW_SQL_ERROR_AT(&partition_attribute_name) + << "The given attribute was not found."; + } + + auto proto = make_unique(); + proto->set_num_partitions(partition_clause->num_partitions()->long_value()); + proto->set_partition_attribute_id(attr_id); + + const std::string partition_type = ToLower(partition_type_string->value()); + if (partition_type == kHashPartitionType) { + proto->set_partition_type(S::PartitionSchemeHeader::HASH); + } else if (partition_type == kRangePartitionType) { + proto->set_partition_type(S::PartitionSchemeHeader::RANGE); + THROW_SQL_ERROR_AT(partition_clause) + << "Range partition is not supported."; + } else { + THROW_SQL_ERROR_AT(partition_type_string) << "Unrecognized partition type: " << partition_type; + } + + return proto.release(); +} + L::LogicalPtr Resolver::resolveCreateIndex( const ParseStatementCreateIndex &create_index_statement) { // Resolve relation reference. http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/resolver/Resolver.hpp ---------------------------------------------------------------------- diff --git a/query_optimizer/resolver/Resolver.hpp b/query_optimizer/resolver/Resolver.hpp index 855e6ba..a3d0833 100644 --- a/query_optimizer/resolver/Resolver.hpp +++ b/query_optimizer/resolver/Resolver.hpp @@ -74,9 +74,8 @@ class PtrList; class StorageBlockLayoutDescription; class Type; -} // namespace quickstep +namespace serialization { class PartitionSchemeHeader; } -namespace quickstep { namespace optimizer { class OptimizerContext; @@ -213,6 +212,16 @@ class Resolver { const ParseStatementCreateTable &create_table_statement); /** + * @brief Resolves the PARTITION clause of a CREATE TABLE statement to a + * the serialized PartitionSchemeHeader describing the user input. + * + * @param create_table_statement The create table statement. + * @return A pointer to a user-owned serialized PartitionSchemeHeader. + */ + const serialization::PartitionSchemeHeader* resolvePartitionClause( + const ParseStatementCreateTable &create_table_statement); + + /** * @brief Resolves a DELETE query and returns a logical plan. * * @param delete_statement The DELETE parse tree. http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/strategy/OneToOne.cpp ---------------------------------------------------------------------- diff --git a/query_optimizer/strategy/OneToOne.cpp b/query_optimizer/strategy/OneToOne.cpp index 78003f4..7d0c4cb 100644 --- a/query_optimizer/strategy/OneToOne.cpp +++ b/query_optimizer/strategy/OneToOne.cpp @@ -121,7 +121,8 @@ bool OneToOne::generatePlan(const L::LogicalPtr &logical_input, std::static_pointer_cast(logical_input); *physical_output = P::CreateTable::Create(create_table->relation_name(), create_table->attributes(), - create_table->block_properties()); + create_table->block_properties(), + create_table->partition_scheme_header_proto()); return true; } case L::LogicalType::kDeleteTuples: { http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/tests/execution_generator/CMakeLists.txt ---------------------------------------------------------------------- diff --git a/query_optimizer/tests/execution_generator/CMakeLists.txt b/query_optimizer/tests/execution_generator/CMakeLists.txt index 1ea6a17..2705130 100644 --- a/query_optimizer/tests/execution_generator/CMakeLists.txt +++ b/query_optimizer/tests/execution_generator/CMakeLists.txt @@ -107,6 +107,11 @@ add_test(quickstep_queryoptimizer_tests_executiongenerator_join "${CMAKE_CURRENT_SOURCE_DIR}/Join.test" "${CMAKE_CURRENT_BINARY_DIR}/Join.test" "${CMAKE_CURRENT_BINARY_DIR}/Join/") +add_test(quickstep_queryoptimizer_tests_executiongenerator_partition + "../quickstep_queryoptimizer_tests_ExecutionGeneratorTest" + "${CMAKE_CURRENT_SOURCE_DIR}/Partition.test" + "${CMAKE_CURRENT_BINARY_DIR}/Partition.test" + "${CMAKE_CURRENT_BINARY_DIR}/Partition/") add_test(quickstep_queryoptimizer_tests_executiongenerator_select "../quickstep_queryoptimizer_tests_ExecutionGeneratorTest" "${CMAKE_CURRENT_SOURCE_DIR}/Select.test" @@ -150,6 +155,7 @@ file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/Drop) file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/Index) file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/Insert) file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/Join) +file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/Partition) file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/Select) file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/StringPatternMatching) file(MAKE_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/TableGenerator) http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/tests/execution_generator/Create.test ---------------------------------------------------------------------- diff --git a/query_optimizer/tests/execution_generator/Create.test b/query_optimizer/tests/execution_generator/Create.test index 5bd0e76..4ffa665 100644 --- a/query_optimizer/tests/execution_generator/Create.test +++ b/query_optimizer/tests/execution_generator/Create.test @@ -40,3 +40,12 @@ CREATE TABLE foo2 (col1 INT, col2 VARCHAR(80)) WITH BLOCKPROPERTIES (TYPE columnstore, SORT col2); -- ERROR: BLOCKPROPERTIES is invalid. +== + +CREATE TABLE foo3 (attr INT) PARTITION BY HASH(attr) PARTITIONS 4; +SELECT * FROM foo3; +-- ++-----------+ +|attr | ++-----------+ ++-----------+ http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/tests/execution_generator/Partition.test ---------------------------------------------------------------------- diff --git a/query_optimizer/tests/execution_generator/Partition.test b/query_optimizer/tests/execution_generator/Partition.test new file mode 100644 index 0000000..ab05391 --- /dev/null +++ b/query_optimizer/tests/execution_generator/Partition.test @@ -0,0 +1,54 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +CREATE TABLE foo (id INT NULL, + name CHAR(20)) +PARTITION BY HASH(id) PARTITIONS 4; + +INSERT INTO foo +SELECT int_col, char_col +FROM test +WHERE int_col > 0 OR int_col < 0; + +SELECT * FROM foo; +-- ++-----------+--------------------+ +|id |name | ++-----------+--------------------+ +| 4| 4 2.000000| +| 8| 8 2.828427| +| 12| 12 3.464102| +| 16| 16 4.000000| +| 24| 24 4.898979| +| -3| -3 1.732051| +| -7| -7 2.645751| +| -11| -11 3.316625| +| -15| -15 3.872983| +| -19| -19 4.358899| +| -23| -23 4.795832| +| 2| 2 1.414214| +| 6| 6 2.449490| +| 14| 14 3.741657| +| 18| 18 4.242641| +| 22| 22 4.690416| +| -1| -1 1.000000| +| -5| -5 2.236068| +| -9| -9 3.000000| +| -13| -13 3.605551| +| -17| -17 4.123106| +| -21| -21 4.582576| ++-----------+--------------------+ http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/tests/logical_generator/Create.test ---------------------------------------------------------------------- diff --git a/query_optimizer/tests/logical_generator/Create.test b/query_optimizer/tests/logical_generator/Create.test index aeff9ec..04134f9 100644 --- a/query_optimizer/tests/logical_generator/Create.test +++ b/query_optimizer/tests/logical_generator/Create.test @@ -46,3 +46,13 @@ TopLevelPlan +-output_attributes= +-AttributeReference[id=0,name=attr,relation=foo,type=Int] +-AttributeReference[id=1,name=attr2,relation=foo,type=Int] +== + +CREATE TABLE foo (attr INT) PARTITION BY HASH(attr) PARTITIONS 4; +-- +TopLevelPlan ++-plan=CreateTable[relation=foo] +| +-attributes= +| +-AttributeReference[id=0,name=attr,relation=foo,type=Int] ++-output_attributes= + +-AttributeReference[id=0,name=attr,relation=foo,type=Int] http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/tests/physical_generator/Create.test ---------------------------------------------------------------------- diff --git a/query_optimizer/tests/physical_generator/Create.test b/query_optimizer/tests/physical_generator/Create.test index 54af3fa..c555371 100644 --- a/query_optimizer/tests/physical_generator/Create.test +++ b/query_optimizer/tests/physical_generator/Create.test @@ -115,3 +115,21 @@ TopLevelPlan | +-AttributeReference[id=0,name=col1,relation=foo,type=Int] +-output_attributes= +-AttributeReference[id=0,name=col1,relation=foo,type=Int] +== + +CREATE TABLE foo (attr INT) PARTITION BY HASH(attr) PARTITIONS 4; +-- +[Optimized Logical Plan] +TopLevelPlan ++-plan=CreateTable[relation=foo] +| +-attributes= +| +-AttributeReference[id=0,name=attr,relation=foo,type=Int] ++-output_attributes= + +-AttributeReference[id=0,name=attr,relation=foo,type=Int] +[Physical Plan] +TopLevelPlan ++-plan=CreateTable[relation=foo] +| +-attributes= +| +-AttributeReference[id=0,name=attr,relation=foo,type=Int] ++-output_attributes= + +-AttributeReference[id=0,name=attr,relation=foo,type=Int] http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/6b063893/query_optimizer/tests/resolver/Create.test ---------------------------------------------------------------------- diff --git a/query_optimizer/tests/resolver/Create.test b/query_optimizer/tests/resolver/Create.test index b04d785..28bd4f5 100644 --- a/query_optimizer/tests/resolver/Create.test +++ b/query_optimizer/tests/resolver/Create.test @@ -229,3 +229,13 @@ CREATE TABLE foo (attr INT) WITH BLOCKPROPERTIES ERROR: The BLOCKSIZEMB property must be between 2MB and 1024MB. (2 : 23) (TYPE split_rowstore, BLOCKSIZEMB 2000); ^ +== + +CREATE TABLE foo (attr INT) PARTITION BY HASH(attr) PARTITIONS 4; +-- +TopLevelPlan ++-plan=CreateTable[relation=foo] +| +-attributes= +| +-AttributeReference[id=0,name=attr,relation=foo,type=Int] ++-output_attributes= + +-AttributeReference[id=0,name=attr,relation=foo,type=Int]