Return-Path: X-Original-To: apmail-hive-dev-archive@www.apache.org Delivered-To: apmail-hive-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 88AE010335 for ; Fri, 14 Nov 2014 00:03:35 +0000 (UTC) Received: (qmail 44377 invoked by uid 500); 14 Nov 2014 00:03:34 -0000 Delivered-To: apmail-hive-dev-archive@hive.apache.org Received: (qmail 44260 invoked by uid 500); 14 Nov 2014 00:03:34 -0000 Mailing-List: contact dev-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list dev@hive.apache.org Received: (qmail 44062 invoked by uid 500); 14 Nov 2014 00:03:34 -0000 Delivered-To: apmail-hadoop-hive-dev@hadoop.apache.org Received: (qmail 44037 invoked by uid 99); 14 Nov 2014 00:03:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 14 Nov 2014 00:03:34 +0000 Date: Fri, 14 Nov 2014 00:03:34 +0000 (UTC) From: "Venki Korukanti (JIRA)" To: hive-dev@hadoop.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-5631) Index creation on a skew table fails MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Venki Korukanti updated HIVE-5631: ---------------------------------- Attachment: HIVE-5631.4.patch.txt Thanks [~ashutoshc] for reviewing the patch. Attached the rebased patch. > Index creation on a skew table fails > ------------------------------------ > > Key: HIVE-5631 > URL: https://issues.apache.org/jira/browse/HIVE-5631 > Project: Hive > Issue Type: Bug > Components: Indexing > Affects Versions: 0.12.0, 0.13.0, 0.14.0 > Reporter: Venki Korukanti > Assignee: Venki Korukanti > Attachments: HIVE-5631.1.patch.txt, HIVE-5631.2.patch.txt, HIVE-5631.3.patch.txt, HIVE-5631.4.patch.txt > > > REPRO STEPS: > create database skewtest; > use skewtest; > create table skew (id bigint, acct string) skewed by (acct) on ('CC','CH'); > create index skew_indx on table skew (id) as 'org.apache.hadoop.hive.ql.index.compact.CompactIndexHandler' WITH DEFERRED REBUILD; > Last DDL fails with following error. > FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. InvalidObjectException(message:Invalid skew column [acct]) > When creating a table, Hive has sanity tests to make sure the columns have proper names and the skewed columns are subset of the table columns. Here we fail because index table has skewed column info. Index tables's skewed columns include {acct} and the columns are {id, _bucketname, _offsets}. As the skewed column {acct} is not part of the table columns Hive throws the exception. > The reason why Index table got skewed column info even though its definition has no such info is: When creating the index table a deep copy of the base table's StorageDescriptor (SD) (in this case 'skew') is made. And in that copied SD, index specific parameters are set and unrelated parameters are reset. Here skewed column info is not reset (there are few other params that are not reset). That's why the index table contains the skewed column info. > Fix: Instead of deep copying the base table StorageDescriptor, create a new one from gathered info. This way it avoids the index table to inherit unnecessary properties in SD from base table. -- This message was sent by Atlassian JIRA (v6.3.4#6332)