Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id C8B6D200BED for ; Sun, 18 Dec 2016 02:34:38 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id C772C160B33; Sun, 18 Dec 2016 01:34:38 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id CC8F9160B39 for ; Sun, 18 Dec 2016 02:34:37 +0100 (CET) Received: (qmail 34621 invoked by uid 500); 18 Dec 2016 01:34:37 -0000 Mailing-List: contact commits-help@quickstep.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@quickstep.incubator.apache.org Delivered-To: mailing list commits@quickstep.incubator.apache.org Received: (qmail 34611 invoked by uid 99); 18 Dec 2016 01:34:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Dec 2016 01:34:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 9B1B81A01FB for ; Sun, 18 Dec 2016 01:34:36 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -6.219 X-Spam-Level: X-Spam-Status: No, score=-6.219 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RCVD_IN_DNSWL_HI=-5, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, RP_MATCHES_RCVD=-2.999] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id 36zLPJxsKiF4 for ; Sun, 18 Dec 2016 01:34:34 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with SMTP id 008665FDB7 for ; Sun, 18 Dec 2016 01:34:30 +0000 (UTC) Received: (qmail 32928 invoked by uid 99); 18 Dec 2016 01:34:30 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 18 Dec 2016 01:34:30 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id F131EDFBAD; Sun, 18 Dec 2016 01:34:29 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: jianqiao@apache.org To: commits@quickstep.incubator.apache.org Date: Sun, 18 Dec 2016 01:34:42 -0000 Message-Id: In-Reply-To: <42ca3e8ce4e24cd4be5de1345ffd3380@git.apache.org> References: <42ca3e8ce4e24cd4be5de1345ffd3380@git.apache.org> X-Mailer: ASF-Git Admin Mailer Subject: [14/42] incubator-quickstep git commit: Refactor regarding the profile flag. archived-at: Sun, 18 Dec 2016 01:34:38 -0000 Refactor regarding the profile flag. Project: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/commit/e7c664ba Tree: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/tree/e7c664ba Diff: http://git-wip-us.apache.org/repos/asf/incubator-quickstep/diff/e7c664ba Branch: refs/heads/exact-filter Commit: e7c664baa216fad0c7d04f08cad8398d6d9db2bd Parents: 7fb96f3 Author: Zuyu Zhang Authored: Fri Nov 18 17:38:17 2016 -0800 Committer: Zuyu Zhang Committed: Sun Nov 20 19:51:25 2016 -0800 ---------------------------------------------------------------------- cli/QuickstepCli.cpp | 13 ++++--------- query_execution/CMakeLists.txt | 3 ++- query_execution/ForemanDistributed.cpp | 6 ++---- query_execution/ForemanDistributed.hpp | 5 +---- query_execution/ForemanSingleNode.cpp | 6 ++---- query_execution/ForemanSingleNode.hpp | 5 +---- query_execution/PolicyEnforcerBase.cpp | 15 +++++++++++++++ query_execution/PolicyEnforcerBase.hpp | 6 +----- query_execution/PolicyEnforcerDistributed.hpp | 5 ++--- query_execution/PolicyEnforcerSingleNode.hpp | 5 ++--- 10 files changed, 32 insertions(+), 37 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/e7c664ba/cli/QuickstepCli.cpp ---------------------------------------------------------------------- diff --git a/cli/QuickstepCli.cpp b/cli/QuickstepCli.cpp index 0e8468e..a4e55d7 100644 --- a/cli/QuickstepCli.cpp +++ b/cli/QuickstepCli.cpp @@ -133,9 +133,6 @@ using tmb::client_id; namespace quickstep { -DEFINE_bool(profile_and_report_workorder_perf, false, - "If true, Quickstep will record the exceution time of all the individual " - "normal work orders and report it at the end of query execution."); DEFINE_int32(num_workers, 0, "Number of worker threads. If this value is " "specified and is greater than 0, then this " "user-supplied value is used. Else (i.e. the" @@ -181,10 +178,9 @@ DEFINE_string(profile_file_name, "", // To put things in perspective, the first run is, in my experiments, about 5-10 // times more expensive than the average run. That means the query needs to be // run at least a hundred times to make the impact of the first run small (< 5 %). -DEFINE_bool(visualize_execution_dag, false, - "If true, visualize the execution plan DAG into a graph in DOT " - "format (DOT is a plain text graph description language) which is " - "then printed via stderr."); + +DECLARE_bool(profile_and_report_workorder_perf); +DECLARE_bool(visualize_execution_dag); } // namespace quickstep @@ -361,8 +357,7 @@ int main(int argc, char* argv[]) { query_processor->getDefaultDatabase(), &storage_manager, -1, // Don't pin the Foreman thread. - num_numa_nodes_system, - quickstep::FLAGS_profile_and_report_workorder_perf || quickstep::FLAGS_visualize_execution_dag); + num_numa_nodes_system); // Start the worker threads. for (Worker &worker : workers) { http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/e7c664ba/query_execution/CMakeLists.txt ---------------------------------------------------------------------- diff --git a/query_execution/CMakeLists.txt b/query_execution/CMakeLists.txt index cf9d5b0..eec0029 100644 --- a/query_execution/CMakeLists.txt +++ b/query_execution/CMakeLists.txt @@ -138,7 +138,8 @@ target_link_libraries(quickstep_queryexecution_PolicyEnforcerBase quickstep_relationaloperators_WorkOrder quickstep_storage_StorageBlockInfo quickstep_utility_Macros - tmb) + tmb + ${GFLAGS_LIB_NAME}) if (ENABLE_DISTRIBUTED) target_link_libraries(quickstep_queryexecution_PolicyEnforcerDistributed glog http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/e7c664ba/query_execution/ForemanDistributed.cpp ---------------------------------------------------------------------- diff --git a/query_execution/ForemanDistributed.cpp b/query_execution/ForemanDistributed.cpp index aa4db17..d619657 100644 --- a/query_execution/ForemanDistributed.cpp +++ b/query_execution/ForemanDistributed.cpp @@ -64,8 +64,7 @@ class QueryHandle; ForemanDistributed::ForemanDistributed( MessageBus *bus, CatalogDatabaseLite *catalog_database, - const int cpu_id, - const bool profile_individual_workorders) + const int cpu_id) : ForemanBase(bus, cpu_id), catalog_database_(DCHECK_NOTNULL(catalog_database)) { const std::vector sender_message_types{ @@ -103,8 +102,7 @@ ForemanDistributed::ForemanDistributed( foreman_client_id_, catalog_database_, &shiftboss_directory_, - bus_, - profile_individual_workorders); + bus_); } void ForemanDistributed::run() { http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/e7c664ba/query_execution/ForemanDistributed.hpp ---------------------------------------------------------------------- diff --git a/query_execution/ForemanDistributed.hpp b/query_execution/ForemanDistributed.hpp index 7f2d2f9..ccdd0ae 100644 --- a/query_execution/ForemanDistributed.hpp +++ b/query_execution/ForemanDistributed.hpp @@ -54,16 +54,13 @@ class ForemanDistributed final : public ForemanBase { * @param bus A pointer to the TMB. * @param catalog_database The catalog database where this query is executed. * @param cpu_id The ID of the CPU to which the Foreman thread can be pinned. - * @param profile_individual_workorders Whether every workorder's execution - * be profiled or not. * * @note If cpu_id is not specified, Foreman thread can be possibly moved * around on different CPUs by the OS. **/ ForemanDistributed(tmb::MessageBus *bus, CatalogDatabaseLite *catalog_database, - const int cpu_id = -1, - const bool profile_individual_workorders = false); + const int cpu_id = -1); ~ForemanDistributed() override {} http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/e7c664ba/query_execution/ForemanSingleNode.cpp ---------------------------------------------------------------------- diff --git a/query_execution/ForemanSingleNode.cpp b/query_execution/ForemanSingleNode.cpp index dfdfb71..02799c7 100644 --- a/query_execution/ForemanSingleNode.cpp +++ b/query_execution/ForemanSingleNode.cpp @@ -63,8 +63,7 @@ ForemanSingleNode::ForemanSingleNode( CatalogDatabaseLite *catalog_database, StorageManager *storage_manager, const int cpu_id, - const size_t num_numa_nodes, - const bool profile_individual_workorders) + const size_t num_numa_nodes) : ForemanBase(bus, cpu_id), main_thread_client_id_(main_thread_client_id), worker_directory_(DCHECK_NOTNULL(worker_directory)), @@ -99,8 +98,7 @@ ForemanSingleNode::ForemanSingleNode( catalog_database_, storage_manager_, worker_directory_, - bus_, - profile_individual_workorders); + bus_); } void ForemanSingleNode::run() { http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/e7c664ba/query_execution/ForemanSingleNode.hpp ---------------------------------------------------------------------- diff --git a/query_execution/ForemanSingleNode.hpp b/query_execution/ForemanSingleNode.hpp index 4cc7a63..d2db51b 100644 --- a/query_execution/ForemanSingleNode.hpp +++ b/query_execution/ForemanSingleNode.hpp @@ -60,8 +60,6 @@ class ForemanSingleNode final : public ForemanBase { * @param storage_manager The StorageManager to use. * @param cpu_id The ID of the CPU to which the Foreman thread can be pinned. * @param num_numa_nodes The number of NUMA nodes in the system. - * @param profile_individual_workorders Whether every workorder's execution - * be profiled or not. * * @note If cpu_id is not specified, Foreman thread can be possibly moved * around on different CPUs by the OS. @@ -72,8 +70,7 @@ class ForemanSingleNode final : public ForemanBase { CatalogDatabaseLite *catalog_database, StorageManager *storage_manager, const int cpu_id = -1, - const std::size_t num_numa_nodes = 1, - const bool profile_individual_workorders = false); + const std::size_t num_numa_nodes = 1); ~ForemanSingleNode() override {} http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/e7c664ba/query_execution/PolicyEnforcerBase.cpp ---------------------------------------------------------------------- diff --git a/query_execution/PolicyEnforcerBase.cpp b/query_execution/PolicyEnforcerBase.cpp index 6e922a8..b799d5f 100644 --- a/query_execution/PolicyEnforcerBase.cpp +++ b/query_execution/PolicyEnforcerBase.cpp @@ -35,10 +35,25 @@ #include "relational_operators/WorkOrder.hpp" #include "storage/StorageBlockInfo.hpp" +#include "gflags/gflags.h" #include "glog/logging.h" namespace quickstep { +DEFINE_bool(profile_and_report_workorder_perf, false, + "If true, Quickstep will record the exceution time of all the individual " + "normal work orders and report it at the end of query execution."); + +DEFINE_bool(visualize_execution_dag, false, + "If true, visualize the execution plan DAG into a graph in DOT " + "format (DOT is a plain text graph description language) which is " + "then printed via stderr."); + +PolicyEnforcerBase::PolicyEnforcerBase(CatalogDatabaseLite *catalog_database) + : catalog_database_(catalog_database), + profile_individual_workorders_(FLAGS_profile_and_report_workorder_perf || FLAGS_visualize_execution_dag) { +} + void PolicyEnforcerBase::processMessage(const TaggedMessage &tagged_message) { std::size_t query_id; QueryManagerBase::dag_node_index op_index; http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/e7c664ba/query_execution/PolicyEnforcerBase.hpp ---------------------------------------------------------------------- diff --git a/query_execution/PolicyEnforcerBase.hpp b/query_execution/PolicyEnforcerBase.hpp index aa070b8..baf9c68 100644 --- a/query_execution/PolicyEnforcerBase.hpp +++ b/query_execution/PolicyEnforcerBase.hpp @@ -54,12 +54,8 @@ class PolicyEnforcerBase { * @brief Constructor. * * @param catalog_database The CatalogDatabase used. - * @param profile_individual_workorders If true, profile each normal work order. **/ - PolicyEnforcerBase(CatalogDatabaseLite *catalog_database, - const bool profile_individual_workorders) - : catalog_database_(catalog_database), - profile_individual_workorders_(profile_individual_workorders) {} + explicit PolicyEnforcerBase(CatalogDatabaseLite *catalog_database); /** * @brief Virtual Destructor. http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/e7c664ba/query_execution/PolicyEnforcerDistributed.hpp ---------------------------------------------------------------------- diff --git a/query_execution/PolicyEnforcerDistributed.hpp b/query_execution/PolicyEnforcerDistributed.hpp index 0bf249c..146e9af 100644 --- a/query_execution/PolicyEnforcerDistributed.hpp +++ b/query_execution/PolicyEnforcerDistributed.hpp @@ -59,9 +59,8 @@ class PolicyEnforcerDistributed final : public PolicyEnforcerBase { PolicyEnforcerDistributed(const tmb::client_id foreman_client_id, CatalogDatabaseLite *catalog_database, ShiftbossDirectory *shiftboss_directory, - tmb::MessageBus *bus, - const bool profile_individual_workorders = false) - : PolicyEnforcerBase(catalog_database, profile_individual_workorders), + tmb::MessageBus *bus) + : PolicyEnforcerBase(catalog_database), foreman_client_id_(foreman_client_id), shiftboss_directory_(shiftboss_directory), bus_(bus) {} http://git-wip-us.apache.org/repos/asf/incubator-quickstep/blob/e7c664ba/query_execution/PolicyEnforcerSingleNode.hpp ---------------------------------------------------------------------- diff --git a/query_execution/PolicyEnforcerSingleNode.hpp b/query_execution/PolicyEnforcerSingleNode.hpp index 16c7a0d..f87d670 100644 --- a/query_execution/PolicyEnforcerSingleNode.hpp +++ b/query_execution/PolicyEnforcerSingleNode.hpp @@ -64,9 +64,8 @@ class PolicyEnforcerSingleNode final : public PolicyEnforcerBase { CatalogDatabaseLite *catalog_database, StorageManager *storage_manager, WorkerDirectory *worker_directory, - tmb::MessageBus *bus, - const bool profile_individual_workorders = false) - : PolicyEnforcerBase(catalog_database, profile_individual_workorders), + tmb::MessageBus *bus) + : PolicyEnforcerBase(catalog_database), foreman_client_id_(foreman_client_id), num_numa_nodes_(num_numa_nodes), storage_manager_(storage_manager),