Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 0E5F6F8C5 for ; Mon, 28 Apr 2014 16:53:36 +0000 (UTC) Received: (qmail 77131 invoked by uid 500); 28 Apr 2014 16:53:32 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 77034 invoked by uid 500); 28 Apr 2014 16:53:32 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 77026 invoked by uid 99); 28 Apr 2014 16:53:32 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Apr 2014 16:53:32 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of dileepkumar.dk@gmail.com designates 209.85.160.170 as permitted sender) Received: from [209.85.160.170] (HELO mail-yk0-f170.google.com) (209.85.160.170) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 28 Apr 2014 16:53:27 +0000 Received: by mail-yk0-f170.google.com with SMTP id 79so3199994ykr.29 for ; Mon, 28 Apr 2014 09:53:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=XZ8N6quwBKXaWBbnyJCXTN2YsZcd8j0JnCQJjCINBGw=; b=sMd+WdKt03Xx8SF2ImbIla7tKB6jqqqr3OVYm3xVN4Vm19dsOSFrj02dM1eLXWV30q m7Ng9TEUiyqxj9bSwZwRkc/VsYZYUKDUsy+x//jcUinybFZrI7AxhJA7uyDbdAfr0erA MTZCqnr4Tc1JGNyU6Jxz6rg0JUzVVqSZUKmUGQXUAe34MBEkixxvC3FnJ49rfYvcx7Ly VxUPgj9l73Q/oMHYWPztHrxMmcqmrOoEl/mONS2jf/vtcnawziPZlufRkAMBsKqydRkd +HknsUymu/VRaxXNeYU/eRWcIiWRC0y9pQXcjaiKsjKmuFxljBU++vzklr+ZORsu9atK rpWA== MIME-Version: 1.0 X-Received: by 10.236.35.201 with SMTP id u49mr4955344yha.98.1398703984369; Mon, 28 Apr 2014 09:53:04 -0700 (PDT) Received: by 10.170.135.3 with HTTP; Mon, 28 Apr 2014 09:53:04 -0700 (PDT) In-Reply-To: References: Date: Mon, 28 Apr 2014 09:53:04 -0700 Message-ID: Subject: Re: analyze hive tables with null values in partition columns From: Dileep Kumar To: user@hive.apache.org Content-Type: multipart/alternative; boundary=20cf3005df6a2264bc04f81d26c9 X-Virus-Checked: Checked by ClamAV on apache.org --20cf3005df6a2264bc04f81d26c9 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable > I have a table that has partition based on column ss_sold_date_sk which > has null value partition as well. > > When I run the analyze ..compute stat it fails with attached exception. > Is there a way to avoid this or bypass this exception, also what would be > the impact on query performance of stat being not collected ? > > The analyze statement I run is as: > > analyze table store_sales partition(ss_sold_date_sk) compute statistics; > Please suggest. > > Thanks, > Dileep > > Diagnostic Messages for this Task: > > Error: java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error whil= e > processing row > {"ss_sold_time_sk":null,"ss_item_sk":null,"ss_customer_sk":null,"ss_cdemo= _sk":null,"ss_hdemo_sk":null,"ss_addr_sk":null,"ss_store_sk":null,"ss_promo= _sk":null,"ss_ticket_number":null,"ss_quantity":null,"ss_wholesale_cost":nu= ll,"ss_list_price":null,"ss_sales_price":null,"ss_ext_discount_amt":null,"s= s_ext_sales_price":null,"ss_ext_wholesale_cost":null,"ss_ext_list_price":nu= ll,"ss_ext_tax":null,"ss_coupon_amt":null,"ss_net_paid":null,"ss_net_paid_i= nc_tax":null,"ss_net_profit":null,"ss_sold_date_sk":null,"raw__data__size":= 0} > > at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195) > > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) > > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) > > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) > > at java.security.AccessController.doPrivileged(Native Method) > > at javax.security.auth.Subject.doAs(Subject.java:415) > > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation= .java:1556) > > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime > Error while processing row > {"ss_sold_time_sk":null,"ss_item_sk":null,"ss_customer_sk":null,"ss_cdemo= _sk":null,"ss_hdemo_sk":null,"ss_addr_sk":null,"ss_store_sk":null,"ss_promo= _sk":null,"ss_ticket_number":null,"ss_quantity":null,"ss_wholesale_cost":nu= ll,"ss_list_price":null,"ss_sales_price":null,"ss_ext_discount_amt":null,"s= s_ext_sales_price":null,"ss_ext_wholesale_cost":null,"ss_ext_list_price":nu= ll,"ss_ext_tax":null,"ss_coupon_amt":null,"ss_net_paid":null,"ss_net_paid_i= nc_tax":null,"ss_net_profit":null,"ss_sold_date_sk":null,"raw__data__size":= 0} > > --20cf3005df6a2264bc04f81d26c9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

=
I have a table that ha= s partition based on column ss_sold_date_sk which has null value partition = as well.

When I run the analyze ..compute stat it fails with att= ached exception.
Is there a way to avoid this or bypass this exception, also what would= be the impact on query performance of stat being not collected ?

The analyze statement I run is as:

analyze tab= le store_sales partition(ss_sold_date_sk) compute statistics;

Please suggest.

Thanks,
Dilee= p

Diagnostic Messages for this Task:

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveE= xception: Hive Runtime Error while processing row {"ss_sold_time_sk&qu= ot;:null,"ss_item_sk":null,"ss_customer_sk":null,"= ss_cdemo_sk":null,"ss_hdemo_sk":null,"ss_addr_sk":= null,"ss_store_sk":null,"ss_promo_sk":null,"ss_tic= ket_number":null,"ss_quantity":null,"ss_wholesale_cost&= quot;:null,"ss_list_price":null,"ss_sales_price":null,&= quot;ss_ext_discount_amt":null,"ss_ext_sales_price":null,&qu= ot;ss_ext_wholesale_cost":null,"ss_ext_list_price":null,&quo= t;ss_ext_tax":null,"ss_coupon_amt":null,"ss_net_paid&qu= ot;:null,"ss_net_paid_inc_tax":null,"ss_net_profit":nul= l,"ss_sold_date_sk":null,"raw__data__size":0}

at= org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:195)

<= p style=3D"font-family:arial,sans-serif;font-size:12.800000190734863px">at = org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)

at= org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)

at org.ap= ache.hadoop.mapred.MapTask.run(MapTask.java:342)

at= org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)

at java.se= curity.AccessController.doPrivileged(Native Method)

at= javax.security.auth.Subject.doAs(Subject.java:415)

at org.apache.hadoop.s= ecurity.UserGroupInformation.doAs(UserGroupInformation.java:1556)

at= org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)

Caused by: or= g.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while pr= ocessing row {"ss_sold_time_sk":null,"ss_item_sk":null,= "ss_customer_sk":null,"ss_cdemo_sk":null,"ss_hdemo= _sk":null,"ss_addr_sk":null,"ss_store_sk":null,&qu= ot;ss_promo_sk":null,"ss_ticket_number":null,"ss_quanti= ty":null,"ss_wholesale_cost":null,"ss_list_price":= null,"ss_sales_price":null,"ss_ext_discount_amt":null,&= quot;ss_ext_sales_price":null,"ss_ext_wholesale_cost":null,&= quot;ss_ext_list_price":null,"ss_ext_tax":null,"ss_coup= on_amt":null,"ss_net_paid":null,"ss_net_paid_inc_tax&qu= ot;:null,"ss_net_profit":null,"ss_sold_date_sk":null,&q= uot;raw__data__size":0}



--20cf3005df6a2264bc04f81d26c9--