Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 686DD200C06 for ; Fri, 13 Jan 2017 02:03:52 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id 66FF1160B40; Fri, 13 Jan 2017 01:03:52 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 652CD160B4C for ; Fri, 13 Jan 2017 02:03:51 +0100 (CET) Received: (qmail 50608 invoked by uid 500); 13 Jan 2017 01:03:50 -0000 Mailing-List: contact commits-help@quickstep.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@quickstep.incubator.apache.org Delivered-To: mailing list commits@quickstep.incubator.apache.org Received: (qmail 50599 invoked by uid 99); 13 Jan 2017 01:03:50 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jan 2017 01:03:50 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 2775CC023D for ; Fri, 13 Jan 2017 01:03:50 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.219 X-Spam-Level: X-Spam-Status: No, score=-6.219 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 9D9lEMt43yrm for ; Fri, 13 Jan 2017 01:03:47 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with SMTP id F0E4D5F4AE for ; Fri, 13 Jan 2017 01:03:46 +0000 (UTC) Received: (qmail 50491 invoked by uid 99); 13 Jan 2017 01:03:46 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 13 Jan 2017 01:03:46 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 6CEFBDFB78; Fri, 13 Jan 2017 01:03:46 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: zuyuz@apache.org To: commits@quickstep.incubator.apache.org Date: Fri, 13 Jan 2017 01:03:47 -0000 Message-Id: <611a0086280543119b0fcb1af14418fd@git.apache.org> In-Reply-To: References: X-Mailer: ASF-Git Admin Mailer Subject: [2/4] incubator-quickstep git commit: Refactored the method to reconstruct PartitionSchemeHeader. archived-at: Fri, 13 Jan 2017 01:03:52 -0000 Refactored the method to reconstruct PartitionSchemeHeader. Project: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/commit/c8c5d45b Tree: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/tree/c8c5d45b Diff: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/diff/c8c5d45b Branch: refs/heads/quickstep_partition_parser_support Commit: c8c5d45bd3553e05ef3f87996925c49f964e8682 Parents: 8d25d43 Author: Zuyu Zhang Authored: Tue Jan 10 14:10:24 2017 -0800 Committer: Zuyu Zhang Committed: Wed Jan 11 13:36:57 2017 -0800 ---------------------------------------------------------------------- catalog/CMakeLists.txt | 3 +++ catalog/Catalog.proto | 3 ++- catalog/CatalogRelation.cpp | 8 ++------ catalog/PartitionScheme.cpp | 5 ++--- catalog/PartitionScheme.hpp | 6 +----- catalog/PartitionSchemeHeader.cpp | 15 +++++++++++++-- catalog/PartitionSchemeHeader.hpp | 7 ++++--- catalog/tests/PartitionScheme_unittest.cpp | 4 ++-- storage/InsertDestination.cpp | 7 +------ 9 files changed, 30 insertions(+), 28 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/c8c5d45b/catalog/CMakeLists.txt ---------------------------------------------------------------------- diff --git a/catalog/CMakeLists.txt b/catalog/CMakeLists.txt index 143d12f..7de9a67 100644 --- a/catalog/CMakeLists.txt +++ b/catalog/CMakeLists.txt @@ -168,6 +168,9 @@ target_link_libraries(quickstep_catalog_PartitionSchemeHeader glog quickstep_catalog_CatalogTypedefs quickstep_catalog_Catalog_proto + quickstep_types_Type + quickstep_types_TypeFactory + quickstep_types_Type_proto quickstep_types_TypedValue quickstep_types_TypedValue_proto quickstep_types_operations_comparisons_Comparison http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/c8c5d45b/catalog/Catalog.proto ---------------------------------------------------------------------- diff --git a/catalog/Catalog.proto b/catalog/Catalog.proto index a54862f..89cb7e5 100644 --- a/catalog/Catalog.proto +++ b/catalog/Catalog.proto @@ -49,7 +49,8 @@ message PartitionSchemeHeader { message RangePartitionSchemeHeader { extend PartitionSchemeHeader { // All required. - repeated TypedValue partition_range_boundaries = 32; + optional Type partition_attr_type = 32; + repeated TypedValue partition_range_boundaries = 33; } } http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/c8c5d45b/catalog/CatalogRelation.cpp ---------------------------------------------------------------------- diff --git a/catalog/CatalogRelation.cpp b/catalog/CatalogRelation.cpp index c288532..41c503c 100644 --- a/catalog/CatalogRelation.cpp +++ b/catalog/CatalogRelation.cpp @@ -43,8 +43,6 @@ namespace quickstep { -class Type; - bool CatalogRelation::ProtoIsValid(const serialization::CatalogRelationSchema &proto) { // Check that proto is fully initialized. if (!CatalogRelationSchema::ProtoIsValid(proto) || @@ -90,11 +88,9 @@ CatalogRelation::CatalogRelation(const serialization::CatalogRelationSchema &pro const serialization::PartitionScheme &proto_partition_scheme = proto.GetExtension(serialization::CatalogRelation::partition_scheme); - const attribute_id partition_attribute_id = proto_partition_scheme.header().partition_attribute_id(); - DCHECK(hasAttributeWithId(partition_attribute_id)); - const Type &attr_type = attr_vec_[partition_attribute_id].getType(); + DCHECK(hasAttributeWithId(proto_partition_scheme.header().partition_attribute_id())); - setPartitionScheme(PartitionScheme::ReconstructFromProto(proto_partition_scheme, attr_type)); + setPartitionScheme(PartitionScheme::ReconstructFromProto(proto_partition_scheme)); // Deserializing the NUMA placement scheme for the relation. #ifdef QUICKSTEP_HAVE_LIBNUMA http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/c8c5d45b/catalog/PartitionScheme.cpp ---------------------------------------------------------------------- diff --git a/catalog/PartitionScheme.cpp b/catalog/PartitionScheme.cpp index d6e3469..1d7dce0 100644 --- a/catalog/PartitionScheme.cpp +++ b/catalog/PartitionScheme.cpp @@ -55,8 +55,7 @@ bool PartitionScheme::ProtoIsValid( return true; } -PartitionScheme* PartitionScheme::ReconstructFromProto(const serialization::PartitionScheme &proto, - const Type &attr_type) { +PartitionScheme* PartitionScheme::ReconstructFromProto(const serialization::PartitionScheme &proto) { DCHECK(ProtoIsValid(proto)) << "Attempted to create PartitionScheme from an invalid proto description:\n" << proto.DebugString(); @@ -74,7 +73,7 @@ PartitionScheme* PartitionScheme::ReconstructFromProto(const serialization::Part } return new PartitionScheme( - PartitionSchemeHeader::ReconstructFromProto(proto.header(), attr_type), + PartitionSchemeHeader::ReconstructFromProto(proto.header()), move(blocks_in_partition)); } http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/c8c5d45b/catalog/PartitionScheme.hpp ---------------------------------------------------------------------- diff --git a/catalog/PartitionScheme.hpp b/catalog/PartitionScheme.hpp index c882a5e..b69a33f 100644 --- a/catalog/PartitionScheme.hpp +++ b/catalog/PartitionScheme.hpp @@ -39,8 +39,6 @@ namespace quickstep { -class Type; - /** \addtogroup Catalog * @{ */ @@ -82,11 +80,9 @@ class PartitionScheme { * * @param proto The Protocol Buffer serialization of a Partition Scheme, * previously produced by getProto(). - * @param attr_type The attribute type of the partitioning attribute. * @return The deserialized partition scheme object. **/ - static PartitionScheme* ReconstructFromProto(const serialization::PartitionScheme &proto, - const Type &attr_type); + static PartitionScheme* ReconstructFromProto(const serialization::PartitionScheme &proto); /** * @brief Check whether a serialization::PartitionScheme is fully-formed and http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/c8c5d45b/catalog/PartitionSchemeHeader.cpp ---------------------------------------------------------------------- diff --git a/catalog/PartitionSchemeHeader.cpp b/catalog/PartitionSchemeHeader.cpp index 26c317d..a4a9063 100644 --- a/catalog/PartitionSchemeHeader.cpp +++ b/catalog/PartitionSchemeHeader.cpp @@ -24,6 +24,9 @@ #include #include "catalog/Catalog.pb.h" +#include "types/Type.hpp" +#include "types/Type.pb.h" +#include "types/TypeFactory.hpp" #include "types/TypedValue.hpp" #include "types/TypedValue.pb.h" @@ -59,7 +62,8 @@ bool PartitionSchemeHeader::ProtoIsValid( case serialization::PartitionSchemeHeader::RANGE: { const std::size_t num_ranges = proto.ExtensionSize(serialization::RangePartitionSchemeHeader::partition_range_boundaries); - return num_ranges == proto.num_partitions() - 1; + return num_ranges == proto.num_partitions() - 1 && + proto.HasExtension(serialization::RangePartitionSchemeHeader::partition_attr_type); } default: // Partition type is unknown. @@ -68,7 +72,7 @@ bool PartitionSchemeHeader::ProtoIsValid( } PartitionSchemeHeader* PartitionSchemeHeader::ReconstructFromProto( - const serialization::PartitionSchemeHeader &proto, const Type &attr_type) { + const serialization::PartitionSchemeHeader &proto) { DCHECK(ProtoIsValid(proto)) << "Attempted to create PartitionSchemeHeader from an invalid proto description:\n" << proto.DebugString(); @@ -78,6 +82,10 @@ PartitionSchemeHeader* PartitionSchemeHeader::ReconstructFromProto( return new HashPartitionSchemeHeader(proto.num_partitions(), proto.partition_attribute_id()); } case serialization::PartitionSchemeHeader::RANGE: { + const Type &attr_type = + TypeFactory::ReconstructFromProto(proto.GetExtension( + serialization::RangePartitionSchemeHeader::partition_attr_type)); + std::vector partition_ranges; for (int i = 0; i < proto.ExtensionSize(serialization::RangePartitionSchemeHeader::partition_range_boundaries); @@ -123,6 +131,9 @@ serialization::PartitionSchemeHeader RangePartitionSchemeHeader::getProto() cons proto.set_num_partitions(num_partitions_); proto.set_partition_attribute_id(partition_attribute_id_); + proto.MutableExtension(serialization::RangePartitionSchemeHeader::partition_attr_type) + ->MergeFrom(partition_attr_type_->getProto()); + for (std::size_t i = 0; i < partition_range_boundaries_.size(); ++i) { proto.AddExtension(serialization::RangePartitionSchemeHeader::partition_range_boundaries) ->MergeFrom(partition_range_boundaries_[i].getProto()); http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/c8c5d45b/catalog/PartitionSchemeHeader.hpp ---------------------------------------------------------------------- diff --git a/catalog/PartitionSchemeHeader.hpp b/catalog/PartitionSchemeHeader.hpp index 89efc6d..c1d65d1 100644 --- a/catalog/PartitionSchemeHeader.hpp +++ b/catalog/PartitionSchemeHeader.hpp @@ -65,12 +65,10 @@ class PartitionSchemeHeader { * * @param proto The Protocol Buffer serialization of a PartitionSchemeHeader, * previously produced by getProto(). - * @param attr_type The attribute type of the partitioning attribute. * @return The reconstructed PartitionSchemeHeader object. **/ static PartitionSchemeHeader* ReconstructFromProto( - const serialization::PartitionSchemeHeader &proto, - const Type &attr_type); + const serialization::PartitionSchemeHeader &proto); /** * @brief Check whether a serialization::PartitionSchemeHeader is fully-formed @@ -227,6 +225,7 @@ class RangePartitionSchemeHeader : public PartitionSchemeHeader { const attribute_id attribute, std::vector &&partition_range) : PartitionSchemeHeader(PartitionType::kRange, num_partitions, attribute), + partition_attr_type_(&partition_attribute_type), partition_range_boundaries_(std::move(partition_range)) { DCHECK_EQ(num_partitions - 1, partition_range_boundaries_.size()); @@ -298,6 +297,8 @@ class RangePartitionSchemeHeader : public PartitionSchemeHeader { } } + const Type* partition_attr_type_; + // The boundaries for each range in the RangePartitionSchemeHeader. // The upper bound of the range is stored here. const std::vector partition_range_boundaries_; http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/c8c5d45b/catalog/tests/PartitionScheme_unittest.cpp ---------------------------------------------------------------------- diff --git a/catalog/tests/PartitionScheme_unittest.cpp b/catalog/tests/PartitionScheme_unittest.cpp index d10b26e..97a9092 100644 --- a/catalog/tests/PartitionScheme_unittest.cpp +++ b/catalog/tests/PartitionScheme_unittest.cpp @@ -511,7 +511,7 @@ TEST(PartitionSchemeTest, CheckHashPartitionSchemeSerialization) { } std::unique_ptr part_scheme_from_proto; part_scheme_from_proto.reset( - PartitionScheme::ReconstructFromProto(part_scheme->getProto(), TypeFactory::GetType(kInt))); + PartitionScheme::ReconstructFromProto(part_scheme->getProto())); const PartitionSchemeHeader &header = part_scheme->getPartitionSchemeHeader(); const PartitionSchemeHeader &header_from_proto = part_scheme_from_proto->getPartitionSchemeHeader(); @@ -561,7 +561,7 @@ TEST(PartitionSchemeTest, CheckRangePartitionSchemeSerialization) { std::unique_ptr part_scheme_from_proto; part_scheme_from_proto.reset( - PartitionScheme::ReconstructFromProto(part_scheme->getProto(), type)); + PartitionScheme::ReconstructFromProto(part_scheme->getProto())); const PartitionSchemeHeader &header = part_scheme->getPartitionSchemeHeader(); const PartitionSchemeHeader &header_from_proto = part_scheme_from_proto->getPartitionSchemeHeader(); http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/c8c5d45b/storage/InsertDestination.cpp ---------------------------------------------------------------------- diff --git a/storage/InsertDestination.cpp b/storage/InsertDestination.cpp index 067edf6..19bb356 100644 --- a/storage/InsertDestination.cpp +++ b/storage/InsertDestination.cpp @@ -56,8 +56,6 @@ using std::vector; namespace quickstep { -class Type; - InsertDestination::InsertDestination(const CatalogRelationSchema &relation, const StorageBlockLayout *layout, StorageManager *storage_manager, @@ -132,11 +130,8 @@ InsertDestination* InsertDestination::ReconstructFromProto( partitions.push_back(move(partition)); } - const serialization::PartitionSchemeHeader &proto_partition_scheme_header = proto_partition_scheme.header(); - const Type &attr_type = - relation.getAttributeById(proto_partition_scheme_header.partition_attribute_id())->getType(); return new PartitionAwareInsertDestination( - PartitionSchemeHeader::ReconstructFromProto(proto_partition_scheme_header, attr_type), + PartitionSchemeHeader::ReconstructFromProto(proto_partition_scheme.header()), relation, layout, storage_manager,