Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 1B2C610B26 for ; Wed, 3 Jul 2013 22:02:37 +0000 (UTC) Received: (qmail 37696 invoked by uid 500); 3 Jul 2013 22:02:35 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 37650 invoked by uid 500); 3 Jul 2013 22:02:35 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 37642 invoked by uid 99); 3 Jul 2013 22:02:35 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Jul 2013 22:02:35 +0000 X-ASF-Spam-Status: No, hits=2.6 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,TRACKER_ID X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of Sanjay.Subramanian@wizecommerce.com designates 216.32.181.185 as permitted sender) Received: from [216.32.181.185] (HELO ch1outboundpool.messaging.microsoft.com) (216.32.181.185) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 03 Jul 2013 22:02:30 +0000 Received: from mail69-ch1-R.bigfish.com (10.43.68.234) by CH1EHSOBE014.bigfish.com (10.43.70.64) with Microsoft SMTP Server id 14.1.225.22; Wed, 3 Jul 2013 22:02:09 +0000 Received: from mail69-ch1 (localhost [127.0.0.1]) by mail69-ch1-R.bigfish.com (Postfix) with ESMTP id 2B6DE1C0494 for ; Wed, 3 Jul 2013 22:02:09 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.232.197;KIP:(null);UIP:(null);IPV:NLI;H:BLUPRD0411HT002.namprd04.prod.outlook.com;RD:none;EFVD:NLI X-SpamScore: 2 X-BigFish: PS2(zzbb2dI9371I936eI181fIdd85k9a6kzz1f42h1ee6h1de0h1fdah2073h1202h1e76h1d1ah1d2ah1fc6hzz18c673h8275bh8275dhz2fh2a8h668h839hbe3he5bhf0ah1288h12a5h12bdh137ah1441h1504h1537h153bh162dh1631h1758h18e1h1946h19b5h1ad9h1b0ah1bceh1d0ch1d2eh1d3fh1dc1h1dfeh1dffh1e1dh1155h) Received-SPF: pass (mail69-ch1: domain of wizecommerce.com designates 157.56.232.197 as permitted sender) client-ip=157.56.232.197; envelope-from=Sanjay.Subramanian@wizecommerce.com; helo=BLUPRD0411HT002.namprd04.prod.outlook.com ;.outlook.com ; Received: from mail69-ch1 (localhost.localdomain [127.0.0.1]) by mail69-ch1 (MessageSwitch) id 1372888927332846_27019; Wed, 3 Jul 2013 22:02:07 +0000 (UTC) Received: from CH1EHSMHS040.bigfish.com (snatpool1.int.messaging.microsoft.com [10.43.68.247]) by mail69-ch1.bigfish.com (Postfix) with ESMTP id 4DEF52E005A for ; Wed, 3 Jul 2013 22:02:07 +0000 (UTC) Received: from BLUPRD0411HT002.namprd04.prod.outlook.com (157.56.232.197) by CH1EHSMHS040.bigfish.com (10.43.69.249) with Microsoft SMTP Server (TLS) id 14.1.225.23; Wed, 3 Jul 2013 22:02:07 +0000 Received: from BLUPRD0411MB426.namprd04.prod.outlook.com ([169.254.10.227]) by BLUPRD0411HT002.namprd04.prod.outlook.com ([10.255.127.37]) with mapi id 14.16.0324.000; Wed, 3 Jul 2013 22:02:06 +0000 From: Sanjay Subramanian To: "user@hive.apache.org" Subject: =?windows-1250?Q?Re:_One_query_works_the_other_does_not=8Aany_clues_=3F?= Thread-Topic: =?windows-1250?Q?One_query_works_the_other_does_not=8Aany_clues_=3F?= Thread-Index: AQHOeDXk4Ux8WvmA8k+EAvY6sl67iplTDCGA Date: Wed, 3 Jul 2013 22:02:05 +0000 Message-ID: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.255.127.4] Content-Type: multipart/alternative; boundary="_000_CDF9ECF09000sanjaysubramanianwizecommercecom_" MIME-Version: 1.0 X-OriginatorOrg: wizecommerce.com X-Virus-Checked: Checked by ClamAV on apache.org --_000_CDF9ECF09000sanjaysubramanianwizecommercecom_ Content-Type: text/plain; charset="windows-1250" Content-Transfer-Encoding: quoted-printable For the time being I have added the create HDFS dir in my hive script=85go= t to keep moving on=85.cant wait for ideal solution :-) but would love to k= now the ideal solution ! !hdfs dfs -mkdir /user/beeswax/warehouse/impressions_hive_stats/outpdir_imp= ressions_header/2013-07-01/record_counts ; INSERT OVERWRITE DIRECTORY '/user/beeswax/warehouse/impressions_hive_stats/= outpdir_impressions_header/2013-07-01/record_counts' SELECT 'outpdir_impressions_header', '2013-07-01', 'record_counts', 'all_servers', count(*) FROM outpdir_impressions_header WHERE header_date_partition=3D'2013-07-01' ; From: Sanjay Subramanian > Reply-To: "user@hive.apache.org" > Date: Wednesday, July 3, 2013 2:40 PM To: "user@hive.apache.org" > Subject: One query works the other does not=8Aany clues ? THIS FAILS =3D=3D=3D=3D=3D=3D=3D=3D=3D INSERT OVERWRITE DIRECTORY '/user/beeswax/warehouse/impressions_hive_stats/= outpdir_impressions_header/2013-07-01/record_counts' select 'outpdir_impres= sions_header', '2013-07-01', 'record_counts', 'all_servers', count(*) from = outpdir_impressions_header where header_date_partition=3D'2013-07-01' 13/07/03 14:28:31 ERROR exec.Task: Failed with exception Unable to rename: = hdfs://thv-nn1.pv.sv.nextag.com:8020/tmp/hive-nextag/hive_2013-07-03_14-25-= 39_324_3631745157890476605/-ext-10000 to: /user/beeswax/warehouse/impressio= ns_hive_stats/outpdir_impressions_header/2013-07-01/record_counts org.apache.hadoop.hive.ql.metadata.HiveException: Unable to rename: hdfs://= thv-nn1.pv.sv.nextag.com:8020/tmp/hive-nextag/hive_2013-07-03_14-25-39_324_= 3631745157890476605/-ext-10000 to: /user/beeswax/warehouse/impressions_hive= _stats/outpdir_impressions_header/2013-07-01/record_counts at org.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:9= 5) at org.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:15= 9) at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:153) at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunn= er.java:57) at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1331) at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1117) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:950) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.j= ava:258) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:2= 15) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:= 406) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:= 341) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:695) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:607) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessor= Impl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethod= AccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:208) FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.= MoveTask THIS SUCCEEDS =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D INSERT OVERWRITE DIRECTORY '/user/beeswax/warehouse/impressions_hive_stats/= outpdir_impressions_header/record_counts/2013-07-01' select 'outpdir_impres= sions_header', '2013-07-01', 'record_counts', 'all_servers', count(*) from = outpdir_impressions_header where header_date_partition=3D'2013-07-01' TABLE DEFINITION =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D CREATE EXTERNAL TABLE IF NOT EXISTS impressions_hive_stats(table_name STRI= NG, aggregation_date STRING , metric_name STRING, metric_key STRING, metric= _value BIGINT) PARTITIONED BY (table_name_partition STRING, aggregation_dat= e_partition STRING , metric_name_partition STRING, metric_key_partition STR= ING) STORED AS INPUTFORMAT \"com.hadoop.mapred.DeprecatedLzoTextInputForm= at\" OUTPUTFORMAT \"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputF= ormat\" ; CONFIDENTIALITY NOTICE =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply ema= il and destroy all copies of the original message along with any attachment= s, from your computer system. If you are the intended recipient, please be = advised that the content of this message is subject to access, review and d= isclosure by the sender's Email System Administrator. CONFIDENTIALITY NOTICE =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply ema= il and destroy all copies of the original message along with any attachment= s, from your computer system. If you are the intended recipient, please be = advised that the content of this message is subject to access, review and d= isclosure by the sender's Email System Administrator. --_000_CDF9ECF09000sanjaysubramanianwizecommercecom_ Content-Type: text/html; charset="windows-1250" Content-ID: Content-Transfer-Encoding: quoted-printable
For the time = being I have added  the create HDFS dir in my hive script=85got to kee= p moving on=85.cant wait for ideal solution :-) but would love to know the = ideal solution !  

!hdfs d= fs -mkdir /user/beeswax/warehouse/impressions_hive_stats/ou= tpdir_impressions_header/2013-07-01/record_counts
;
INSERT = OVERWRITE DIRECTORY '/user/beeswax/warehouse/impressions_hive_stats/outpdir= _impressions_header/2013-07-01/record_counts' 
SELECT<= /span>
  =    'outpdir_impressions_header', 
  =    '2013-07-01', 
  =    'record_counts', 
  =    'all_servers', 
  =    count(*) 
FROM&nb= sp;
  =    outpdir_impressions_header 
WHERE
  =    header_date_partition=3D'2013-07-01'
;

From: Sanjay Subramanian <sanjay.subramanian@wizecom= merce.com>
Reply-To: "user@hive.apache.org" <user@hive.apache.org>
Date: Wednesday, July 3, 2013 2:40 = PM
To: "user@hive.apache.org" <user@hive.apache.org>
Subject: One query works the other = does not=8Aany clues ?

THIS FAILS
=3D=3D=3D=3D=3D=3D=3D=3D=3D
INSERT OVERWRITE DIRECTORY '/user/beeswax/w= arehouse/impressions_hive_stats/outpdir_impressions_header/2013-07-01/re= cord_counts' select 'outpdir_impressions_header', '2013-07-01', 'record_counts', 'all_servers', count(*) from outpdir_impres= sions_header where header_date_partition=3D'2013-07-01'

13/07/03 14:28:31 ERROR exec.Task: Failed w= ith exception Unable to rename: hdfs://thv-nn1.pv.sv.nextag.com:8020/tmp/hi= ve-nextag/hive_2013-07-03_14-25-39_324_3631745157890476605/-ext-10000 to: /= user/beeswax/warehouse/impressions_hive_stats/outpdir_impressions_header/20= 13-07-01/record_counts
org.apache.hadoop.hive.ql.metadata.HiveExce= ption: Unable to rename: hdfs://thv-nn1.pv.sv.nextag.com:8020/tmp/hive-next= ag/hive_2013-07-03_14-25-39_324_3631745157890476605/-ext-10000 to: /user/be= eswax/warehouse/impressions_hive_stats/outpdir_impressions_header/2013-07-0= 1/record_counts
        at org.apache.h= adoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:95)
        at org.apache.h= adoop.hive.ql.exec.MoveTask.execute(MoveTask.java:159)
        at org.apache.h= adoop.hive.ql.exec.Task.executeTask(Task.java:153)
        at org.apache.h= adoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
        at org.apache.h= adoop.hive.ql.Driver.launchTask(Driver.java:1331)
        at org.apache.h= adoop.hive.ql.Driver.execute(Driver.java:1117)
        at org.apache.h= adoop.hive.ql.Driver.run(Driver.java:950)
        at org.apache.h= adoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:258)
        at org.apache.h= adoop.hive.cli.CliDriver.processCmd(CliDriver.java:215)
        at org.apache.h= adoop.hive.cli.CliDriver.processLine(CliDriver.java:406)
        at org.apache.h= adoop.hive.cli.CliDriver.processLine(CliDriver.java:341)
        at org.apache.h= adoop.hive.cli.CliDriver.run(CliDriver.java:695)
        at org.apache.h= adoop.hive.cli.CliDriver.main(CliDriver.java:607)
        at sun.reflect.= NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.= NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.= DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.re= flect.Method.invoke(Method.java:597)
        at org.apache.h= adoop.util.RunJar.main(RunJar.java:208)

FAILED: Execution Error, return code 1 from= org.apache.hadoop.hive.ql.exec.MoveTask



THIS SUCCEEDS
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
INSERT OVERWRITE DIRECTORY '/user/beeswax/warehouse/impressions_hive= _stats/outpdir_impressions_header/record_counts/2013-07-01' select '= outpdir_impressions_header', '2013-07-01', 'record_counts', 'all_servers', count(*) from outpdir_impressions_header w= here header_date_partition=3D'2013-07-01'

TABLE DEFINITION
=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
CREATE&= nbsp;EXTERNAL&= nbsp;TABLE  IF NOT EXISTS impressions_hive_stats(table_name STRING, aggregation_date STRING , metric_name STRING, metric_key STRING, metric_valu= e BIGINT) PARTITIONED BY (table_name_partition STRING, aggregation_date_partition STRING , metric_name_partition STRING, metric_key_partit= ion STRING)  STORED&nbs= p;AS INPUTFORMAT  \"com.hadoop.mapred.DeprecatedLzoTextInputFormat\"  &= nbsp;OUTPUTFORMAT \"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutp= utFormat\" ;

CONFIDENTIALITY NOTICE
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply email and destroy all copies of the ori= ginal message along with any attachments, from your computer system. If you= are the intended recipient, please be advised that the content of this mes= sage is subject to access, review and disclosure by the sender's Email System Administrator.

CONFIDENTIALITY NOTICE
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply email and destroy all copies of the ori= ginal message along with any attachments, from your computer system. If you= are the intended recipient, please be advised that the content of this mes= sage is subject to access, review and disclosure by the sender's Email System Administrator.
--_000_CDF9ECF09000sanjaysubramanianwizecommercecom_--