Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 583A2DD75 for ; Thu, 26 Jul 2012 17:18:43 +0000 (UTC) Received: (qmail 20699 invoked by uid 500); 26 Jul 2012 17:18:42 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 20649 invoked by uid 500); 26 Jul 2012 17:18:42 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 20640 invoked by uid 99); 26 Jul 2012 17:18:41 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jul 2012 17:18:41 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=FSL_RCVD_USER,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of ablimit@gmail.com designates 209.85.214.176 as permitted sender) Received: from [209.85.214.176] (HELO mail-ob0-f176.google.com) (209.85.214.176) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jul 2012 17:18:34 +0000 Received: by obbtb18 with SMTP id tb18so3784414obb.35 for ; Thu, 26 Jul 2012 10:18:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=iRsdtnGqeqZbCTVDeYfQpgJJ430J3bGKDfZlhT1xNMc=; b=A/6ROYEuEZClSJoEbPXszOkQd6tmcgEL8m07NZxaJdh/3oI2YP8B8G5a5QKX4p73Uj cI0crd4wHcuG5QfYd7zKyytHnxeV+5EwXU1+iFoN9bmVX2UxYcPN/U6Ci5ErNqrBE3fT TjxWHUQ8OjjGZb/uTMHwvxT4LYSpf+sPC7LLlA/ojfO5LgxITaplLdiAWQrvsDD8+ubo UZuOjZ2WBkQVtqXcbSJ7Z+g4/JlRuOYP4ZJcjNxJNRNhuhr7sxmTeJg9gJ4ZuLTKWLCR xOGZdYtjdqjxQd/i6Kw8g4ryUQyBJsLbT3JKPCUp4lQphMl/wz7nLL70cA24cbzGLs0S c9qA== MIME-Version: 1.0 Received: by 10.60.22.5 with SMTP id z5mr42414496oee.2.1343323093314; Thu, 26 Jul 2012 10:18:13 -0700 (PDT) Received: by 10.182.45.131 with HTTP; Thu, 26 Jul 2012 10:18:13 -0700 (PDT) Date: Thu, 26 Jul 2012 13:18:13 -0400 Message-ID: Subject: Problem with indexing in Hive From: Ablimit Aji To: user@hive.apache.org Content-Type: multipart/alternative; boundary=e89a8fb1f2cacbaa6b04c5bec7ff --e89a8fb1f2cacbaa6b04c5bec7ff Content-Type: text/plain; charset=UTF-8 I have written a custom index handler and wanted to test it. However hive is not using it. So I test with simple table (pokes (int foo, string bar)) which comes with hive distribution for testing purpose. Then I created a compact index and set the set hive.optimize.index.filter=true; However, upon checking the log info, it seems hive is still not using the index. So, what is the problem ? The query I issued is as follow: select foo from pokes WHERE foo=498 ; Below is the log info I got after issuing the query. 12/07/26 12:25:17 INFO index.IndexWhereProcessor: Processing predicate for index optimization 12/07/26 12:25:17 INFO index.IndexWhereProcessor: (foo = 498) 12/07/26 12:25:17 INFO metastore.HiveMetaStore: 0: get_table : db=default tbl=pokes_idx 12/07/26 12:25:17 INFO hive.log: DDL: struct pokes_idx { i32 foo, string _bucketname, list _offsets} 12/07/26 12:25:17 INFO index.IndexWhereProcessor: checking index staleness... 12/07/26 12:25:17 INFO index.IndexWhereProcessor: 1342465077455 12/07/26 12:25:17 INFO index.IndexWhereProcessor: 1342465077455 12/07/26 12:25:17 INFO util.NativeCodeLoader: Loaded the native-hadoop library 12/07/26 12:25:17 WARN snappy.LoadSnappy: Snappy native library not loaded --e89a8fb1f2cacbaa6b04c5bec7ff Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable I have written a custom index handler and wanted to test i= t. However hive is not using it.=C2=A0
So I tes= t with simple table (pokes (int foo, string bar)) which comes with hive dis= tribution for testing purpose.=C2=A0
Then I created a compact index and set the set hive.o= ptimize.index.filter=3Dtrue;=C2=A0
Howeve= r, upon checking the log info, it seems hive is still not using the index.= =C2=A0
So, what is the problem ?=C2=A0
The query I issued is as follow: =C2=A0select foo from pokes W= HERE foo=3D498 ;

Below is the log info I got after issuing the query= .

12/07/26 12:25:17 INFO index.IndexWhereProcessor: Processing predi= cate for index optimization=C2=A0
12/07/26 12:25:17=C2=A0INFO index.IndexWhereProcessor: (foo =3D 498)
12/07/26 12:25:17 INFO metastore.HiveMetaStore: 0: get_table = : db=3Ddefault tbl=3Dpokes_idx
12/07/26 12:25:17 INFO hive.log: DDL: struct pokes_idx { i32 foo, string _b= ucketname, list _offsets}=C2=A0
12/07/26 = 12:25:17 INFO index.IndexWhereProcessor: checking index staleness...
12/= 07/26 12:25:17 INFO index.IndexWhereProcessor: 1342465077455
12/07/26 12:25:17 INFO index.IndexWhereProcessor: 1342465077455
12/07/26= 12:25:17 INFO util.NativeCodeLoader: Loaded the native-hadoop library
1= 2/07/26 12:25:17 WARN snappy.LoadSnappy: Snappy native library not loaded
--e89a8fb1f2cacbaa6b04c5bec7ff--