Return-Path: X-Original-To: apmail-hawq-commits-archive@minotaur.apache.org Delivered-To: apmail-hawq-commits-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 82BA018910 for ; Fri, 25 Dec 2015 04:02:19 +0000 (UTC) Received: (qmail 33992 invoked by uid 500); 25 Dec 2015 04:02:19 -0000 Delivered-To: apmail-hawq-commits-archive@hawq.apache.org Received: (qmail 33937 invoked by uid 500); 25 Dec 2015 04:02:19 -0000 Mailing-List: contact commits-help@hawq.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hawq.incubator.apache.org Delivered-To: mailing list commits@hawq.incubator.apache.org Received: (qmail 33927 invoked by uid 99); 25 Dec 2015 04:02:19 -0000 Received: from Unknown (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Dec 2015 04:02:19 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 983C81A065A for ; Fri, 25 Dec 2015 04:02:18 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.247 X-Spam-Level: * X-Spam-Status: No, score=1.247 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, KAM_LAZY_DOMAIN_SECURITY=1, RP_MATCHES_RCVD=-0.554, URIBL_BLOCKED=0.001] autolearn=disabled Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Y0s2Ftmaiw7z for ; Fri, 25 Dec 2015 04:02:09 +0000 (UTC) Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with SMTP id C16B020CB7 for ; Fri, 25 Dec 2015 04:02:08 +0000 (UTC) Received: (qmail 33920 invoked by uid 99); 25 Dec 2015 04:02:07 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 25 Dec 2015 04:02:07 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id BA692DFC77; Fri, 25 Dec 2015 04:02:07 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: hubertzhang@apache.org To: commits@hawq.incubator.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: incubator-hawq git commit: HAWQ-278. Add min_cost for each query. Date: Fri, 25 Dec 2015 04:02:07 +0000 (UTC) Repository: incubator-hawq Updated Branches: refs/heads/master 454945f42 -> 10e2b18d2 HAWQ-278. Add min_cost for each query. Project: http://git-wip-us.apache.org/repos/asf/incubator-hawq/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-hawq/commit/10e2b18d Tree: http://git-wip-us.apache.org/repos/asf/incubator-hawq/tree/10e2b18d Diff: http://git-wip-us.apache.org/repos/asf/incubator-hawq/diff/10e2b18d Branch: refs/heads/master Commit: 10e2b18d2099474b6a60119650b9b0931bf6e5d5 Parents: 454945f Author: hubertzhang Authored: Fri Dec 25 12:00:47 2015 +0800 Committer: hubertzhang Committed: Fri Dec 25 12:00:47 2015 +0800 ---------------------------------------------------------------------- src/backend/cdb/cdbdatalocality.c | 7 ++++++- src/backend/cdb/cdbvars.c | 1 + src/backend/utils/misc/guc.c | 10 ++++++++++ src/include/cdb/cdbvars.h | 1 + 4 files changed, 18 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/10e2b18d/src/backend/cdb/cdbdatalocality.c ---------------------------------------------------------------------- diff --git a/src/backend/cdb/cdbdatalocality.c b/src/backend/cdb/cdbdatalocality.c index 5c3cbde..3ab90b3 100644 --- a/src/backend/cdb/cdbdatalocality.c +++ b/src/backend/cdb/cdbdatalocality.c @@ -4111,8 +4111,13 @@ calculate_planner_segment_num(Query *query, QueryResourceLife resourceLife, minTargetSegmentNumber = enforce_virtual_segment_number; } uint64_t before_rm_allocate_resource = gettime_microsec(); + + /* cost is use by RM to balance workload between hosts. the cost is at least one block size*/ + int64 mincost = min_cost_for_each_query; + mincost <<= 20; + int64 queryCost = context.total_size < mincost ? mincost : context.total_size; if (QRL_NONE != resourceLife) { - resource = AllocateResource(QRL_ONCE, sliceNum, context.total_size, + resource = AllocateResource(QRL_ONCE, sliceNum, queryCost, maxTargetSegmentNumber, minTargetSegmentNumber, context.host_context.hostnameVolInfos, context.host_context.size); } http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/10e2b18d/src/backend/cdb/cdbvars.c ---------------------------------------------------------------------- diff --git a/src/backend/cdb/cdbvars.c b/src/backend/cdb/cdbvars.c index e9021ee..078957f 100644 --- a/src/backend/cdb/cdbvars.c +++ b/src/backend/cdb/cdbvars.c @@ -267,6 +267,7 @@ int max_filecount_notto_split_segment; int min_datasize_to_combine_segment; int datalocality_algorithm_version; int hash_to_random_flag; +int min_cost_for_each_query; bool metadata_cache_enable; int metadata_cache_block_capacity; http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/10e2b18d/src/backend/utils/misc/guc.c ---------------------------------------------------------------------- diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index e51417f..c4b81c5 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -6273,6 +6273,16 @@ static struct config_int ConfigureNamesInt[] = }, { + {"min_cost_for_each_query", PGC_USERSET, DEVELOPER_OPTIONS, + gettext_noop("Sets min cost(MB) for each query which is utilized by RM"), + NULL, + GUC_NO_SHOW_ALL | GUC_NOT_IN_SAMPLE + }, + &min_cost_for_each_query, + 128, 1, 1024 ,NULL, NULL + }, + + { {"datalocality_algorithm_version", PGC_USERSET, DEVELOPER_OPTIONS, gettext_noop("Sets dalocality algorithm version default is 1(For DEBUG)"), NULL, http://git-wip-us.apache.org/repos/asf/incubator-hawq/blob/10e2b18d/src/include/cdb/cdbvars.h ---------------------------------------------------------------------- diff --git a/src/include/cdb/cdbvars.h b/src/include/cdb/cdbvars.h index 6526168..1cc9764 100644 --- a/src/include/cdb/cdbvars.h +++ b/src/include/cdb/cdbvars.h @@ -1219,6 +1219,7 @@ extern int rm_nresqueue_limit; extern int max_filecount_notto_split_segment; extern int min_datasize_to_combine_segment; extern int datalocality_algorithm_version; +extern int min_cost_for_each_query; typedef enum {