Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id A7368200C8F for ; Thu, 25 May 2017 20:10:08 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id A428E160BB4; Thu, 25 May 2017 18:10:08 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id F354B160BCA for ; Thu, 25 May 2017 20:10:07 +0200 (CEST) Received: (qmail 89784 invoked by uid 500); 25 May 2017 18:10:06 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 89764 invoked by uid 99); 25 May 2017 18:10:06 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 25 May 2017 18:10:06 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 693F6C00B6 for ; Thu, 25 May 2017 18:10:06 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -100.002 X-Spam-Level: X-Spam-Status: No, score=-100.002 tagged_above=-999 required=6.31 tests=[RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id F1w0Hxb-fvZR for ; Thu, 25 May 2017 18:10:05 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 71AE25F477 for ; Thu, 25 May 2017 18:10:05 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id E9D99E0D95 for ; Thu, 25 May 2017 18:10:04 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 49B8821B5C for ; Thu, 25 May 2017 18:10:04 +0000 (UTC) Date: Thu, 25 May 2017 18:10:04 +0000 (UTC) From: "Remus Rusanu (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-16757) Use memoization in HiveRelMdRowCount.getRowCount MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 archived-at: Thu, 25 May 2017 18:10:08 -0000 [ https://issues.apache.org/jira/browse/HIVE-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025141#comment-16025141 ] Remus Rusanu commented on HIVE-16757: ------------------------------------- From what I can see, this seems to be a problem in Calcite code as well. I will start a discussion on calcite-dev. > Use memoization in HiveRelMdRowCount.getRowCount > ------------------------------------------------ > > Key: HIVE-16757 > URL: https://issues.apache.org/jira/browse/HIVE-16757 > Project: Hive > Issue Type: Bug > Components: Query Planning > Reporter: Remus Rusanu > Assignee: Remus Rusanu > Attachments: HIVE-16757.01.patch, HIVE-16757.02.patch > > > On complex queries HiveRelMdRowCount.getRowCount can get called many times. since it does not memoize its result and the call is recursive, it results in an explosion of calls. for example a query with 49 joins, during join ordering (LoptOtimizerJoinRule) the HiveRelMdRowCount.getRowCount gets called 6442 as a top level call, but the recursivity exploded this to 501729 calls. Memoization of the rezult would stop the recursion early. In my testing this reduced the join reordering time for said query from 11s to <1s.. -- This message was sent by Atlassian JIRA (v6.3.15#6346)