Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 25A37200C80 for ; Thu, 25 May 2017 15:01:11 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 2415B160BCA; Thu, 25 May 2017 13:01:11 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 6BF77160BC7 for ; Thu, 25 May 2017 15:01:10 +0200 (CEST) Received: (qmail 79060 invoked by uid 500); 25 May 2017 13:01:09 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 79051 invoked by uid 99); 25 May 2017 13:01:09 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 May 2017 13:01:09 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 4D551CFA0B for ; Thu, 25 May 2017 13:01:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id fwTJo7S0GPlb for ; Thu, 25 May 2017 13:01:05 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id BA2545FCCD for ; Thu, 25 May 2017 13:01:04 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 51DA8E01D8 for ; Thu, 25 May 2017 13:01:04 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 07F992193C for ; Thu, 25 May 2017 13:01:04 +0000 (UTC) Date: Thu, 25 May 2017 13:01:04 +0000 (UTC) From: "Remus Rusanu (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-16757) Use memoization in HiveRelMdRowCount.getRowCount MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 25 May 2017 13:01:11 -0000 [ https://issues.apache.org/jira/browse/HIVE-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Remus Rusanu updated HIVE-16757: -------------------------------- Attachment: HIVE-16757.01.patch Patch 01 for running the tests > Use memoization in HiveRelMdRowCount.getRowCount > ------------------------------------------------ > > Key: HIVE-16757 > URL: https://issues.apache.org/jira/browse/HIVE-16757 > Project: Hive > Issue Type: Bug > Components: Query Planning > Reporter: Remus Rusanu > Assignee: Remus Rusanu > Attachments: HIVE-16757.01.patch > > > On complex queries HiveRelMdRowCount.getRowCount can get called many times. since it does not memoize its result and the call is recursive, it results in an explosion of calls. for example a query with 49 joins, during join ordering (LoptOtimizerJoinRule) the HiveRelMdRowCount.getRowCount gets called 6442 as a top level call, but the recursivity exploded this to 501729 calls. Memoization of the rezult would stop the recursion early. In my testing this reduced the join reordering time for said query from 11s to <1s.. -- This message was sent by Atlassian JIRA (v6.3.15#6346)