Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 30C80200B64 for ; Tue, 2 Aug 2016 11:01:04 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 2F812160A8C; Tue, 2 Aug 2016 09:01:04 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 50362160A76 for ; Tue, 2 Aug 2016 11:01:03 +0200 (CEST) Received: (qmail 96422 invoked by uid 500); 2 Aug 2016 09:01:02 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 96412 invoked by uid 99); 2 Aug 2016 09:01:02 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Aug 2016 09:01:02 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 9794AC6721 for ; Tue, 2 Aug 2016 09:01:01 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.18 X-Spam-Level: * X-Spam-Status: No, score=1.18 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, MIME_QP_LONG_LINE=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id x0ZxHQZnPw9O for ; Tue, 2 Aug 2016 09:00:59 +0000 (UTC) Received: from mail-wm0-f65.google.com (mail-wm0-f65.google.com [74.125.82.65]) by mx2-lw-us.apache.org (ASF Mail Server at mx2-lw-us.apache.org) with ESMTPS id 689215FBBE for ; Tue, 2 Aug 2016 09:00:59 +0000 (UTC) Received: by mail-wm0-f65.google.com with SMTP id i5so29597680wmg.2 for ; Tue, 02 Aug 2016 02:00:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-transfer-encoding:from:mime-version:subject:message-id:date :references:in-reply-to:to; bh=Qb53Dt1hlt3eCyt2ZxvAtfU1efxG9A4QSA1qu/9EuDw=; b=bHf5WaXHtAS620wDt4EaMRQWGar5WVGrxoR8xz/4oa4rAmZg2PCn7jI/4QFxUC5JZo XfBweYORiOUgmxrGioP2IkVdV3aDwhxUnPF2oVUVOCCrtAZASkdR0Da4O4KfBqNEYO2G uaQGfbVkfTHWf0fARkH6/7jtPtHOd+3ofeqQvheu/XDnD38/Ztft8F3ECKvVOE5qUQTl uicRXyn9sKWGftX4zZxApCMSqD/hNC86geFO2L0jWBkJqlX6eQ+UAwiYPsKJc/2vdna4 71BEt1jpv2Q/8Wjkl86tq0nHq4HhZsfJVnPeDen5yHxa2n5ZE4hn0tgeNZrjEUmz1+PH rybw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:content-transfer-encoding:from:mime-version :subject:message-id:date:references:in-reply-to:to; bh=Qb53Dt1hlt3eCyt2ZxvAtfU1efxG9A4QSA1qu/9EuDw=; b=Yx605fT3ZB+qBLKqD16Hp+W2TV8tzxT2wWPKte98vbBfSaAkENufzLz70uEiE9Ys17 5JeYwKnr7YREKaCRluctVOIqFke0GfGTjPbyMP4oPL72Yt55vQW4a98kMyFaewhTKYLy JwzpRmLnwsOojltwe81XUCdJfE59DxFwNuRy2WUn2bS70NsNcWneYnUay/ryqhEcOugW fPwy8FHtkYgCN4Dko0ErPxju16cPUoY5vCLAZlSS3HBbAvO3Am6lwigO3NxIdq2rRbuR 0iPt4qDjJ3f/ha1Y9T00DBwiDgeQrvFr8qvHY2lJueQOt67xgjaEdZTcXKS5V0i1gqQw GM7w== X-Gm-Message-State: AEkooutJIcx/r7YlQx+HozWN0d6XLAX0BChT//Svr+qge2brM1WptyShrdyOPJy1HgzUSQ== X-Received: by 10.194.192.195 with SMTP id hi3mr53705608wjc.108.1470128458170; Tue, 02 Aug 2016 02:00:58 -0700 (PDT) Received: from [10.208.193.173] ([176.0.92.193]) by smtp.gmail.com with ESMTPSA id gg10sm1567706wjd.4.2016.08.02.02.00.57 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Tue, 02 Aug 2016 02:00:57 -0700 (PDT) Content-Type: multipart/alternative; boundary=Apple-Mail-FBD893EC-1266-42A5-A28E-F87367F26B30 Content-Transfer-Encoding: 7bit From: =?utf-8?Q?J=C3=B6rn_Franke?= Mime-Version: 1.0 (1.0) Subject: Re: Doubt on Hive Partitioning. Message-Id: Date: Tue, 2 Aug 2016 09:55:25 +0200 References: <4d6c0f1f99384965860bb0d54fe95f12@XORPUN-MBX01.India.XoriantCorp.com> <1F22A0A4-8242-47F7-8E0A-C062FC0BF9CF@gmail.com> In-Reply-To: To: user@hive.apache.org X-Mailer: iPhone Mail (13G34) archived-at: Tue, 02 Aug 2016 09:01:04 -0000 --Apple-Mail-FBD893EC-1266-42A5-A28E-F87367F26B30 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I do not think so, but never tested it. > On 02 Aug 2016, at 03:45, Qiuzhuang Lian wrote:= >=20 > Is this partition pruning fixed in MR too except for TEZ in newer hive ver= sion? >=20 > Regards, > Q >=20 >> On Mon, Aug 1, 2016 at 8:48 PM, J=C3=B6rn Franke w= rote: >> It happens in old hive version of the filter is only in the where clause a= nd NOT in the join clause. This should not happen in newer hive version. You= can check it by executing explain dependency query.=20 >>=20 >>> On 01 Aug 2016, at 11:07, Abhishek Dubey wr= ote: >>>=20 >>> Hi All, >>>=20 >>> =20 >>>=20 >>> I have a very big table t with billions of rows and it is partitioned on= a column p. Column p has datatype text and values like =E2=80=98201601=E2=80= =99, =E2=80=98201602=E2=80=99=E2=80=A6upto =E2=80=98201612=E2=80=99. >>>=20 >>> And, I am running a query like : Select columns from t where p=3D=E2=80=99= 201604=E2=80=99. >>>=20 >>> =20 >>>=20 >>> My question is : Can there be a scenario/condition/probability that my q= uery will do a complete table scan on t instead of only reading data for spe= cified partition key. If yes, please put some light on those scenario. >>>=20 >>> =20 >>>=20 >>> I=E2=80=99m asking this because someone told me that there is a probabil= ity that the query will ignore the partitioning and do a complete table scan= to fetch output. >>>=20 >>> =20 >>>=20 >>>=20 >>> Thanks & Regards, >>> Abhishek Dubey >=20 --Apple-Mail-FBD893EC-1266-42A5-A28E-F87367F26B30 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable
I do not think so, but neve= r tested it.

On 02 Aug 2016, at 03:45, Qiuzhuang Lian <qiuzhuang.lian@gmail.com> wro= te:

Is this par= tition pruning fixed in MR too except for TEZ in newer hive version?
Regards,
Q

<= div class=3D"gmail_quote">On Mon, Aug 1, 2016 at 8:48 PM, J=C3=B6rn Franke <= span dir=3D"ltr"><jornfranke@gmail.com> wrote:
It happens in old hive version of the f= ilter is only in the where clause and NOT in the join clause. This should no= t happen in newer hive version. You can check it by executing explain depend= ency query. 

On 01 Aug 2016, at 11= :07, Abhishek Dubey <Abhishek.Dubey@Xoriant.Com> wrote:

Hi All= ,

 

I have= a very big table t with billions of rows and it is partitioned on a column p. C= olumn p  has datatype text and values like =E2=80=98201601=E2=80=99, =E2= =80=98201602=E2=80=99=E2=80=A6upto =E2=80=98201612=E2=80=99.

And, I= am running a query like : Select columns from t where p=3D=E2=80=99201604=E2=80=99.

=  

My que= stion is : Can there be a scenario/condition/probability that my query will d= o a complete table scan on t instead of only reading data for specified partition key. If yes, p= lease put some light on those scenario.

 

I=E2=80=99m asking this= because someone told me that there is a probability that the query will ign= ore the partitioning and do a complete table scan to fetch output.

 

Tha= nks & Regards,=
Abhishek Dubey <= /p>

 


= --Apple-Mail-FBD893EC-1266-42A5-A28E-F87367F26B30--