Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 7B6DE200D52 for ; Fri, 17 Nov 2017 20:40:47 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 79C01160BFB; Fri, 17 Nov 2017 19:40:47 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 57456160C0A for ; Fri, 17 Nov 2017 20:40:46 +0100 (CET) Received: (qmail 33688 invoked by uid 500); 17 Nov 2017 19:40:45 -0000 Mailing-List: contact commits-help@quickstep.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@quickstep.incubator.apache.org Delivered-To: mailing list commits@quickstep.incubator.apache.org Received: (qmail 33679 invoked by uid 99); 17 Nov 2017 19:40:45 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Nov 2017 19:40:45 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id BAACCC691B for ; Fri, 17 Nov 2017 19:40:44 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -4.222 X-Spam-Level: X-Spam-Status: No, score=-4.222 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id kC7GtKf4HQsu for ; Fri, 17 Nov 2017 19:40:43 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with SMTP id ABAE15FC43 for ; Fri, 17 Nov 2017 19:40:42 +0000 (UTC) Received: (qmail 33468 invoked by uid 99); 17 Nov 2017 19:40:42 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 17 Nov 2017 19:40:42 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 70C8CF5F49; Fri, 17 Nov 2017 19:40:41 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: jianqiao@apache.org To: commits@quickstep.incubator.apache.org Date: Fri, 17 Nov 2017 19:40:41 -0000 Message-Id: <1bc6a760c4df4d25adf68610c1012a38@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [01/27] incubator-quickstep git commit: Printed out the partition info in QueryPlan. [Forced Update!] archived-at: Fri, 17 Nov 2017 19:40:47 -0000 Repository: incubator-quickstep Updated Branches: refs/heads/trace e6c99562b -> aec7623aa (forced update) Printed out the partition info in QueryPlan. Project: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/commit/475704ec Tree: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/tree/475704ec Diff: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/diff/475704ec Branch: refs/heads/trace Commit: 475704ec9510793a70e149b938d02569611c6177 Parents: d85f7a9 Author: Zuyu Zhang Authored: Mon Sep 11 11:57:10 2017 -0500 Committer: Zuyu Zhang Committed: Thu Sep 14 13:15:10 2017 -0500 ---------------------------------------------------------------------- query_optimizer/CMakeLists.txt | 1 + query_optimizer/OptimizerTree.hpp | 39 ++++++++++++++++++++ query_optimizer/logical/CreateTable.cpp | 5 +++ query_optimizer/logical/CreateTable.hpp | 20 ++++++---- query_optimizer/physical/CreateTable.cpp | 5 +++ query_optimizer/physical/CreateTable.hpp | 20 ++++++---- .../tests/logical_generator/Create.test | 15 +++++++- .../tests/physical_generator/Create.test | 26 +++++++++++++ query_optimizer/tests/resolver/Create.test | 17 +++++++-- 9 files changed, 128 insertions(+), 20 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/475704ec/query_optimizer/CMakeLists.txt ---------------------------------------------------------------------- diff --git a/query_optimizer/CMakeLists.txt b/query_optimizer/CMakeLists.txt index 4ea21b2..5e0db44 100644 --- a/query_optimizer/CMakeLists.txt +++ b/query_optimizer/CMakeLists.txt @@ -212,6 +212,7 @@ target_link_libraries(quickstep_queryoptimizer_OptimizerContext quickstep_queryoptimizer_expressions_ExprId quickstep_utility_Macros) target_link_libraries(quickstep_queryoptimizer_OptimizerTree + quickstep_catalog_Catalog_proto quickstep_storage_StorageBlockLayout_proto quickstep_utility_Macros quickstep_utility_TreeStringSerializable) http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/475704ec/query_optimizer/OptimizerTree.hpp ---------------------------------------------------------------------- diff --git a/query_optimizer/OptimizerTree.hpp b/query_optimizer/OptimizerTree.hpp index 62df66d..c54ce20 100644 --- a/query_optimizer/OptimizerTree.hpp +++ b/query_optimizer/OptimizerTree.hpp @@ -25,6 +25,7 @@ #include #include +#include "catalog/Catalog.pb.h" #include "storage/StorageBlockLayout.pb.h" #include "utility/Macros.hpp" #include "utility/TreeStringSerializable.hpp" @@ -283,6 +284,44 @@ OptimizerProtoRepresentation* getOptimizerRepresentationForProto( return node.release(); } +template +OptimizerProtoRepresentation* getOptimizerRepresentationForProto( + const serialization::PartitionSchemeHeader *partition_header) { + if (partition_header == nullptr) { + return nullptr; + } + + auto node = std::make_unique>(); + + // Add properties based on the partition type. + switch (partition_header->partition_type()) { + case serialization::PartitionSchemeHeader::HASH: { + node->addProperty("partition_type", "hash"); + break; + } + case serialization::PartitionSchemeHeader::RANDOM: { + node->addProperty("partition_type", "random"); + break; + } + case serialization::PartitionSchemeHeader::RANGE: { + node->addProperty("partition_type", "range"); + // TODO(quickstep-team): display the range boundaries. + node->addProperty("range_boundaries", "TODO"); + break; + } + default: + LOG(FATAL) << "Unrecognized partition type in protobuf message."; + } + // Every case will specify a partition number and a partition attributes. + node->addProperty("num_partitions", partition_header->num_partitions()); + + for (int i = 0; i < partition_header->partition_attribute_ids_size(); ++i) { + node->addProperty("partition_attr_id", partition_header->partition_attribute_ids(i)); + } + + return node.release(); +} + /** @} */ } // namespace optimizer http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/475704ec/query_optimizer/logical/CreateTable.cpp ---------------------------------------------------------------------- diff --git a/query_optimizer/logical/CreateTable.cpp b/query_optimizer/logical/CreateTable.cpp index 111d04b..9977a0f 100644 --- a/query_optimizer/logical/CreateTable.cpp +++ b/query_optimizer/logical/CreateTable.cpp @@ -49,6 +49,11 @@ void CreateTable::getFieldStringItems( non_container_child_field_names->push_back("block_properties"); non_container_child_fields->push_back(block_properties_representation_); } + + if (partition_scheme_header_proto_representation_) { + non_container_child_field_names->push_back("partition_scheme_header"); + non_container_child_fields->push_back(partition_scheme_header_proto_representation_); + } } } // namespace logical http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/475704ec/query_optimizer/logical/CreateTable.hpp ---------------------------------------------------------------------- diff --git a/query_optimizer/logical/CreateTable.hpp b/query_optimizer/logical/CreateTable.hpp index da4325d..b380ac9 100644 --- a/query_optimizer/logical/CreateTable.hpp +++ b/query_optimizer/logical/CreateTable.hpp @@ -48,7 +48,7 @@ typedef std::shared_ptr CreateTablePtr; /** * @brief Represents an operation that creates a new table. */ -class CreateTable : public Logical { +class CreateTable final : public Logical { public: LogicalType getLogicalType() const override { return LogicalType::kCreateTable; } @@ -138,13 +138,17 @@ class CreateTable : public Logical { block_properties_(block_properties), block_properties_representation_( getOptimizerRepresentationForProto(block_properties_.get())), - partition_scheme_header_proto_(partition_scheme_header_proto) {} - - std::string relation_name_; - std::vector attributes_; - std::shared_ptr block_properties_; - std::shared_ptr > block_properties_representation_; - std::shared_ptr partition_scheme_header_proto_; + partition_scheme_header_proto_(partition_scheme_header_proto), + partition_scheme_header_proto_representation_( + getOptimizerRepresentationForProto(partition_scheme_header_proto_.get())) {} + + const std::string relation_name_; + const std::vector attributes_; + const std::shared_ptr block_properties_; + const std::shared_ptr> block_properties_representation_; + const std::shared_ptr partition_scheme_header_proto_; + const std::shared_ptr> + partition_scheme_header_proto_representation_; DISALLOW_COPY_AND_ASSIGN(CreateTable); }; http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/475704ec/query_optimizer/physical/CreateTable.cpp ---------------------------------------------------------------------- diff --git a/query_optimizer/physical/CreateTable.cpp b/query_optimizer/physical/CreateTable.cpp index d42eac3..95e6a9d 100644 --- a/query_optimizer/physical/CreateTable.cpp +++ b/query_optimizer/physical/CreateTable.cpp @@ -49,6 +49,11 @@ void CreateTable::getFieldStringItems( non_container_child_field_names->push_back("block_properties"); non_container_child_fields->push_back(block_properties_representation_); } + + if (partition_scheme_header_proto_representation_) { + non_container_child_field_names->push_back("partition_scheme_header"); + non_container_child_fields->push_back(partition_scheme_header_proto_representation_); + } } } // namespace physical http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/475704ec/query_optimizer/physical/CreateTable.hpp ---------------------------------------------------------------------- diff --git a/query_optimizer/physical/CreateTable.hpp b/query_optimizer/physical/CreateTable.hpp index 05eab0d..6ebdfa2 100644 --- a/query_optimizer/physical/CreateTable.hpp +++ b/query_optimizer/physical/CreateTable.hpp @@ -49,7 +49,7 @@ typedef std::shared_ptr CreateTablePtr; /** * @brief Creates a table. */ -class CreateTable : public Physical { +class CreateTable final : public Physical { public: PhysicalType getPhysicalType() const override { return PhysicalType::kCreateTable; @@ -145,13 +145,17 @@ class CreateTable : public Physical { block_properties_(block_properties), block_properties_representation_( getOptimizerRepresentationForProto(block_properties_.get())), - partition_scheme_header_proto_(partition_scheme_header_proto) {} - - std::string relation_name_; - std::vector attributes_; - std::shared_ptr block_properties_; - std::shared_ptr > block_properties_representation_; - std::shared_ptr partition_scheme_header_proto_; + partition_scheme_header_proto_(partition_scheme_header_proto), + partition_scheme_header_proto_representation_( + getOptimizerRepresentationForProto(partition_scheme_header_proto_.get())) {} + + const std::string relation_name_; + const std::vector attributes_; + const std::shared_ptr block_properties_; + const std::shared_ptr> block_properties_representation_; + const std::shared_ptr partition_scheme_header_proto_; + const std::shared_ptr> + partition_scheme_header_proto_representation_; DISALLOW_COPY_AND_ASSIGN(CreateTable); }; http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/475704ec/query_optimizer/tests/logical_generator/Create.test ---------------------------------------------------------------------- diff --git a/query_optimizer/tests/logical_generator/Create.test b/query_optimizer/tests/logical_generator/Create.test index aac49fb..89b4a99 100644 --- a/query_optimizer/tests/logical_generator/Create.test +++ b/query_optimizer/tests/logical_generator/Create.test @@ -52,6 +52,10 @@ CREATE TABLE foo (attr INT) PARTITION BY HASH(attr) PARTITIONS 4; -- TopLevelPlan +-plan=CreateTable[relation=foo] +| +-partition_scheme_header=ProtoDescription +| | +-Property=ProtoProperty[Property=partition_type,Value=hash] +| | +-Property=ProtoProperty[Property=num_partitions,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=0] | +-attributes= | +-AttributeReference[id=0,name=attr,relation=foo,type=Int] +-output_attributes= @@ -59,10 +63,19 @@ TopLevelPlan == CREATE TABLE foo (attr1 INT, attr2 LONG, attr3 FLOAT, attr4 DOUBLE, attr5 CHAR(5), attr6 VARCHAR(4)) -PARTITION BY HASH(attr1, attr2, attr3, attr4, attr5, attr6) PARTITIONS 4; +PARTITION BY HASH(attr2, attr1, attr3, attr4, attr5, attr6) PARTITIONS 4; -- TopLevelPlan +-plan=CreateTable[relation=foo] +| +-partition_scheme_header=ProtoDescription +| | +-Property=ProtoProperty[Property=partition_type,Value=hash] +| | +-Property=ProtoProperty[Property=num_partitions,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=1] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=0] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=2] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=3] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=5] | +-attributes= | +-AttributeReference[id=0,name=attr1,relation=foo,type=Int] | +-AttributeReference[id=1,name=attr2,relation=foo,type=Long] http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/475704ec/query_optimizer/tests/physical_generator/Create.test ---------------------------------------------------------------------- diff --git a/query_optimizer/tests/physical_generator/Create.test b/query_optimizer/tests/physical_generator/Create.test index 161cc00..a2dd5c0 100644 --- a/query_optimizer/tests/physical_generator/Create.test +++ b/query_optimizer/tests/physical_generator/Create.test @@ -122,6 +122,10 @@ CREATE TABLE foo (attr INT) PARTITION BY HASH(attr) PARTITIONS 4; [Optimized Logical Plan] TopLevelPlan +-plan=CreateTable[relation=foo] +| +-partition_scheme_header=ProtoDescription +| | +-Property=ProtoProperty[Property=partition_type,Value=hash] +| | +-Property=ProtoProperty[Property=num_partitions,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=0] | +-attributes= | +-AttributeReference[id=0,name=attr,relation=foo,type=Int] +-output_attributes= @@ -129,6 +133,10 @@ TopLevelPlan [Physical Plan] TopLevelPlan +-plan=CreateTable[relation=foo] +| +-partition_scheme_header=ProtoDescription +| | +-Property=ProtoProperty[Property=partition_type,Value=hash] +| | +-Property=ProtoProperty[Property=num_partitions,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=0] | +-attributes= | +-AttributeReference[id=0,name=attr,relation=foo,type=Int] +-output_attributes= @@ -141,6 +149,15 @@ PARTITION BY HASH(attr1, attr2, attr3, attr4, attr5, attr6) PARTITIONS 4; [Optimized Logical Plan] TopLevelPlan +-plan=CreateTable[relation=foo] +| +-partition_scheme_header=ProtoDescription +| | +-Property=ProtoProperty[Property=partition_type,Value=hash] +| | +-Property=ProtoProperty[Property=num_partitions,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=0] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=1] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=2] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=3] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=5] | +-attributes= | +-AttributeReference[id=0,name=attr1,relation=foo,type=Int] | +-AttributeReference[id=1,name=attr2,relation=foo,type=Long] @@ -158,6 +175,15 @@ TopLevelPlan [Physical Plan] TopLevelPlan +-plan=CreateTable[relation=foo] +| +-partition_scheme_header=ProtoDescription +| | +-Property=ProtoProperty[Property=partition_type,Value=hash] +| | +-Property=ProtoProperty[Property=num_partitions,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=0] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=1] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=2] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=3] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=5] | +-attributes= | +-AttributeReference[id=0,name=attr1,relation=foo,type=Int] | +-AttributeReference[id=1,name=attr2,relation=foo,type=Long] http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/475704ec/query_optimizer/tests/resolver/Create.test ---------------------------------------------------------------------- diff --git a/query_optimizer/tests/resolver/Create.test b/query_optimizer/tests/resolver/Create.test index 1372cf4..c216c85 100644 --- a/query_optimizer/tests/resolver/Create.test +++ b/query_optimizer/tests/resolver/Create.test @@ -181,9 +181,7 @@ BLOCKPROPERTIES (TYPE compresse... CREATE TABLE foo (attr INT, attr2 INT) WITH BLOCKPROPERTIES (TYPE compressed_rowstore, COMPRESS 1); -- -ERROR: The COMPRESS property must be specified as ALL or a list of attributes. (2 : 1) -BLOCKPROPERTIES (TYPE compresse... -^ +[same as above] == # All specified COMPRESS columns must exist. @@ -235,6 +233,10 @@ CREATE TABLE foo (attr INT) PARTITION BY HASH(attr) PARTITIONS 4; -- TopLevelPlan +-plan=CreateTable[relation=foo] +| +-partition_scheme_header=ProtoDescription +| | +-Property=ProtoProperty[Property=partition_type,Value=hash] +| | +-Property=ProtoProperty[Property=num_partitions,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=0] | +-attributes= | +-AttributeReference[id=0,name=attr,relation=foo,type=Int] +-output_attributes= @@ -246,6 +248,15 @@ PARTITION BY HASH(attr1, attr2, attr3, attr4, attr5, attr6) PARTITIONS 4; -- TopLevelPlan +-plan=CreateTable[relation=foo] +| +-partition_scheme_header=ProtoDescription +| | +-Property=ProtoProperty[Property=partition_type,Value=hash] +| | +-Property=ProtoProperty[Property=num_partitions,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=0] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=1] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=2] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=3] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=4] +| | +-Property=ProtoProperty[Property=partition_attr_id,Value=5] | +-attributes= | +-AttributeReference[id=0,name=attr1,relation=foo,type=Int] | +-AttributeReference[id=1,name=attr2,relation=foo,type=Long]