Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 808C3200B64 for ; Tue, 2 Aug 2016 13:45:16 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 7F161160A8C; Tue, 2 Aug 2016 11:45:16 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A1AA5160A76 for ; Tue, 2 Aug 2016 13:45:15 +0200 (CEST) Received: (qmail 2222 invoked by uid 500); 2 Aug 2016 11:45:14 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 2212 invoked by uid 99); 2 Aug 2016 11:45:14 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2016 11:45:14 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id D2C3DC0B66 for ; Tue, 2 Aug 2016 11:45:13 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.18 X-Spam-Level: * X-Spam-Status: No, score=1.18 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 9wj-gDluOZur for ; Tue, 2 Aug 2016 11:45:11 +0000 (UTC) Received: from mail-wm0-f49.google.com (mail-wm0-f49.google.com [74.125.82.49]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id 3AE8360D7A for ; Tue, 2 Aug 2016 11:45:11 +0000 (UTC) Received: by mail-wm0-f49.google.com with SMTP id q128so405798505wma.1 for ; Tue, 02 Aug 2016 04:45:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-transfer-encoding:mime-version:subject:message-id:date :references:in-reply-to:to; bh=xTPdnGLgJK8PWXybDhi2qEe6MReStnYKOJPZNesbNa0=; b=J1s8sPoaJA9a6NXsukkkQAmK/B9NMsioSDfL5x3ejNZwGoWQbrigYTEgvmbHOhmwWj Qy6tXcWZsUSg7vibhCclsxxILqh9UN9Q7TLgRDnaausEN3yvUsZJh6g5tXzeCyIP7L/p orGJl+hJq/2vIPlGOw8MQGnmMLr9YIPbkG0+3Ekb9wHrjLPWVJwnCowz5J8dlYmKGMNi dHoBE32zXKPArjxoJbLdktl6k61psLoFcBHNY+lrG6LZ62v0/wUmbkQlt/00cUBGlVvB RgE5QQQUAP2bSgdWvIQD2WCHpt7JYHOnOUXumwP+vqHIsm2KMEWT/NhTSOsqS9YxuDWF 42Wg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:references:in-reply-to:to; bh=xTPdnGLgJK8PWXybDhi2qEe6MReStnYKOJPZNesbNa0=; b=KgQNJEHLZICy9S0blx9V0YgfbMxMk/O15xLzPVDdNxHNDyvbXXRtrpqJT6wr2aiX9K xtaYxYJAVAnT38cpfkxUuL9mXGyxUYYyDe8egnpIJg9lmn5O08XVtZa/Da3PMVbtU29t cBcxb4Jh4Maefb/gTzt27ioyUv6VSMwXS92W38GXthGV+Re636uL0RbmdCmY8Tmiz8W4 gKVnII4qdiYN0sTCl6ZMo3JLuLnvuip3qoaLNBOOTlJi+KFlbKVwshywsNeRVBRme9pM WIv4UQTVEPmZx4xGuMrnPQAYNGuEayyz87q7tse0rmsiRn53ZiJkz/4ZwV8XvpITeQme OBxg== X-Gm-Message-State: AEkoousxwMKoZm6d61SaJKMj3Xuu27fggaqtwAtghR35HNC+ZUjN9etAQ1cR1DlNqRWogw== X-Received: by 10.28.170.198 with SMTP id t189mr18275350wme.19.1470138310726; Tue, 02 Aug 2016 04:45:10 -0700 (PDT) Received: from [10.208.193.173] ([176.0.92.193]) by smtp.gmail.com with ESMTPSA id f3sm2271753wjh.2.2016.08.02.04.45.03 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 02 Aug 2016 04:45:09 -0700 (PDT) From: =?utf-8?Q?J=C3=B6rn_Franke?= Content-Type: multipart/alternative; boundary=Apple-Mail-A83C654B-AB55-4F6A-9AD7-4AEFBEAEEBE6 Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (1.0) Subject: Re: Doubt on Hive Partitioning. Message-Id: Date: Tue, 2 Aug 2016 13:43:49 +0200 References: <4d6c0f1f99384965860bb0d54fe95f12@XORPUN-MBX01.India.XoriantCorp.com> <1F22A0A4-8242-47F7-8E0A-C062FC0BF9CF@gmail.com> In-Reply-To: To: user@hive.apache.org X-Mailer: iPhone Mail (13G34) archived-at: Tue, 02 Aug 2016 11:45:16 -0000 --Apple-Mail-A83C654B-AB55-4F6A-9AD7-4AEFBEAEEBE6 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Partition pruning works also with older Hive version, but you have to put th= e filter in the join statement and not only in the where statement=20 > On 02 Aug 2016, at 09:53, Furcy Pin wrote: >=20 > I'm using Hive 1.1 on MR and dynamic partition pruning does not seem to wo= rk. >=20 > Since MR is deprecated in 2.0, I assume we should not expect any future pe= rf optimisation on this side. >=20 > It has been implemented for Hive on Spark, though. > https://issues.apache.org/jira/browse/HIVE-9152 >=20 >=20 >=20 >=20 >> On Tue, Aug 2, 2016 at 3:45 AM, Qiuzhuang Lian = wrote: >> Is this partition pruning fixed in MR too except for TEZ in newer hive ve= rsion? >>=20 >> Regards, >> Q >>=20 >>> On Mon, Aug 1, 2016 at 8:48 PM, J=C3=B6rn Franke w= rote: >>> It happens in old hive version of the filter is only in the where clause= and NOT in the join clause. This should not happen in newer hive version. Y= ou can check it by executing explain dependency query.=20 >>>=20 >>>> On 01 Aug 2016, at 11:07, Abhishek Dubey w= rote: >>>>=20 >>>> Hi All, >>>>=20 >>>> =20 >>>>=20 >>>> I have a very big table t with billions of rows and it is partitioned o= n a column p. Column p has datatype text and values like =E2=80=98201601=E2= =80=99, =E2=80=98201602=E2=80=99=E2=80=A6upto =E2=80=98201612=E2=80=99. >>>>=20 >>>> And, I am running a query like : Select columns from t where p=3D=E2=80= =99201604=E2=80=99. >>>>=20 >>>> =20 >>>>=20 >>>> My question is : Can there be a scenario/condition/probability that my q= uery will do a complete table scan on t instead of only reading data for spe= cified partition key. If yes, please put some light on those scenario. >>>>=20 >>>> =20 >>>>=20 >>>> I=E2=80=99m asking this because someone told me that there is a probabi= lity that the query will ignore the partitioning and do a complete table sca= n to fetch output. >>>>=20 >>>> =20 >>>>=20 >>>>=20 >>>> Thanks & Regards, >>>> Abhishek Dubey >=20 --Apple-Mail-A83C654B-AB55-4F6A-9AD7-4AEFBEAEEBE6 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
Partition pruning works als= o with older Hive version, but you have to put the filter in the join statem= ent and not only in the where statement 

On 02 Aug 2016, a= t 09:53, Furcy Pin <furcy.pin@f= laminem.com> wrote:

I'm using Hive 1.1 on MR and dynamic partition pruning does not s= eem to work.

Since MR is deprecated in 2.0, I assume we should n= ot expect any future perf optimisation on this side.

It has been implemented for Hive on Spark, though.



On Tu= e, Aug 2, 2016 at 3:45 AM, Qiuzhuang Lian <qiuzhuang.lian@gmail.com> wrote:
Is thi= s partition pruning fixed in MR too except for TEZ in newer hive version?
Regards,
Q

On Mon= , Aug 1, 2016 at 8:48 PM, J=C3=B6rn Franke <jornfranke@gmail.com><= /span> wrote:
=
It happens in old hive version of the filter is only in the where claus= e and NOT in the join clause. This should not happen in newer hive version. Y= ou can check it by executing explain dependency query. 
=

On 01 Aug 2016, at 11:07, Abhishek Dubey <Abhishek.Dubey@Xoriant.Com>= wrote:

Hi All= ,

 

I have= a very big table t with billions of rows and it is partitioned on a column p. C= olumn p  has datatype text and values like =E2=80=98201601=E2=80=99, =E2= =80=98201602=E2=80=99=E2=80=A6upto =E2=80=98201612=E2=80=99.

And, I= am running a query like : Select columns from t where p=3D=E2=80=99201604=E2=80=99.

=  

My que= stion is : Can there be a scenario/condition/probability that my query will d= o a complete table scan on t instead of only reading data for specified partition key. If yes, p= lease put some light on those scenario.

 

I=E2=80=99m asking this= because someone told me that there is a probability that the query will ign= ore the partitioning and do a complete table scan to fetch output.

 

Tha= nks & Regards,=
Abhishek Dubey <= /p>

 



= --Apple-Mail-A83C654B-AB55-4F6A-9AD7-4AEFBEAEEBE6--