Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0C8B217A77 for ; Thu, 22 Jan 2015 02:14:35 +0000 (UTC) Received: (qmail 44154 invoked by uid 500); 22 Jan 2015 02:14:34 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 44079 invoked by uid 500); 22 Jan 2015 02:14:34 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 44068 invoked by uid 500); 22 Jan 2015 02:14:34 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 44065 invoked by uid 99); 22 Jan 2015 02:14:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 22 Jan 2015 02:14:34 +0000 Date: Thu, 22 Jan 2015 02:14:34 +0000 (UTC) From: "Ashutosh Chauhan (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (HIVE-9341) Apply ColumnPrunning for noop PTFs MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-9341?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14286795#comment-14286795 ] Ashutosh Chauhan commented on HIVE-9341: ---------------------------------------- can you also update RB with test changes. easier to review that way. > Apply ColumnPrunning for noop PTFs > ---------------------------------- > > Key: HIVE-9341 > URL: https://issues.apache.org/jira/browse/HIVE-9341 > Project: Hive > Issue Type: Improvement > Components: PTF-Windowing > Reporter: Navis > Assignee: Navis > Priority: Trivial > Attachments: HIVE-9341.1.patch.txt, HIVE-9341.2.patch.txt, HIVE-9341.3.patch.txt, HIVE-9341.4.patch.txt > > > Currently, PTF disables CP optimization, which can make a huge burden. For example, > {noformat} > select p_mfgr, p_name, p_size, > rank() over (partition by p_mfgr order by p_name) as r, > dense_rank() over (partition by p_mfgr order by p_name) as dr, > sum(p_retailprice) over (partition by p_mfgr order by p_name rows between unbounded preceding and current row) as s1 > from noop(on part > partition by p_mfgr > order by p_name > ); > STAGE PLANS: > Stage: Stage-1 > Map Reduce > Map Operator Tree: > TableScan > alias: part > Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE > Reduce Output Operator > key expressions: p_mfgr (type: string), p_name (type: string) > sort order: ++ > Map-reduce partition columns: p_mfgr (type: string) > Statistics: Num rows: 26 Data size: 3147 Basic stats: COMPLETE Column stats: NONE > value expressions: p_partkey (type: int), p_name (type: string), p_mfgr (type: string), p_brand (type: string), p_type (type: string), p_size (type: int), p_container (type: string), p_retailprice (type: double), p_comment (type: string), BLOCK__OFFSET__INSIDE__FILE (type: bigint), INPUT__FILE__NAME (type: string), ROW__ID (type: struct) > ... > {noformat} > There should be a generic way to discern referenced columns but before that, we know CP can be safely applied to noop functions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)