Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id CE5B210F91 for ; Mon, 29 Jul 2013 05:09:38 +0000 (UTC) Received: (qmail 68918 invoked by uid 500); 29 Jul 2013 05:09:37 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 68517 invoked by uid 500); 29 Jul 2013 05:09:32 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 68509 invoked by uid 99); 29 Jul 2013 05:09:31 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Jul 2013 05:09:31 +0000 X-ASF-Spam-Status: No, hits=4.1 required=5.0 tests=FREEMAIL_REPLY,HK_RANDOM_ENVFROM,HK_RANDOM_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of huaiyin.thu@gmail.com designates 209.85.128.49 as permitted sender) Received: from [209.85.128.49] (HELO mail-qe0-f49.google.com) (209.85.128.49) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 29 Jul 2013 05:09:25 +0000 Received: by mail-qe0-f49.google.com with SMTP id 1so2052767qec.8 for ; Sun, 28 Jul 2013 22:09:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=Lc7Neho8dsonco/6w0wCBovxtJTRMzUcfgRPlkkGZTM=; b=z+UuqKh/COzVJv3TSZfZCeThKs/VABpFGUAndPZm29q6XzpRMyZ/jdoXXY7ecN1cud 5TZO1GuZMeQA2wSRFEiBwH8/oWgzAPTF0IWdqyUnJlPqRR+e21RS3fPJLPlvR3Uix7f3 Om+/IwWP7ybCk1TrOxh2qQSFtvJ7+AWwxjeJNxnUjmwCEnK3jNNGB2eakpOUUEUyrKbE ZqdlTiBtmOz4O6B8K8ul6RlI/X2XKI/yLmIu/z1XotBbf/7PiCPyBDJHpedtqhYwT/yU irwyuqQq34CRnALxWDFPexBmMLfnaEtLhsJC0DHdeiKzHPEsdqjnmHhhJ8soMbO+gj0k 3HfQ== X-Received: by 10.224.126.129 with SMTP id c1mr28151723qas.93.1375074544106; Sun, 28 Jul 2013 22:09:04 -0700 (PDT) MIME-Version: 1.0 Received: by 10.49.129.105 with HTTP; Sun, 28 Jul 2013 22:08:44 -0700 (PDT) In-Reply-To: <8A5A27F7EDA74F03921ECA1C19503D49@gmail.com> References: <8A5A27F7EDA74F03921ECA1C19503D49@gmail.com> From: Yin Huai Date: Sun, 28 Jul 2013 22:08:44 -0700 Message-ID: Subject: =?UTF-8?B?UmU6IOWbnuWkje+8miBCVUcgSU4gSElWRS00NjUwIHNlZW1zIG5vdCBmaXhlZA==?= To: user@hive.apache.org Content-Type: multipart/alternative; boundary=001a11c2e328bdc8b604e29f7d42 X-Virus-Checked: Checked by ClamAV on apache.org --001a11c2e328bdc8b604e29f7d42 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hi, Can you also post the output of EXPLAIN? The execution plan may be helpful to locate the problem. Thanks, Yin On Sun, Jul 28, 2013 at 8:06 PM, wrote: > What I mean by "not pass the testcase in HIVE-4650" is that I compile the > trunk code and run the query in HIVE-4650: > SELECT * > FROM > (SELECT VAL001 x1, > VAL002 x2, > VAL003 x3, > VAL004 x4, > VAL005 y > FROM > (SELECT /*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1) VAL001,(VAL002- > mu2) * 1/(sd2) VAL002,(VAL003- mu3) * 1/(sd3) VAL003,(VAL004- mu4) * > 1/(sd4) VAL004,(VAL005- mu5) * 1/(sd5) VAL005 > FROM > (SELECT * > FROM > (SELECT x1 VAL001, > x2 VAL002, > x3 VAL003, > x4 VAL004, > y VAL005 > FROM cmnt) obj1_3) v3 > JOIN > (SELECT count(*) c, > avg(VAL001) mu1, > avg(VAL002) mu2, > avg(VAL003) mu3, > avg(VAL004) mu4, > avg(VAL005) mu5, > stddev_pop(VAL001) sd1, > stddev_pop(VAL002) sd2, > stddev_pop(VAL003) sd3, > stddev_pop(VAL004) sd4, > stddev_pop(VAL005) sd5 > FROM > (SELECT * > FROM > (SELECT x1 VAL001, > x2 VAL002, > x3 VAL003, > x4 VAL004, > y VAL005 > FROM cmnt) obj1_3) v1) v2) obj1_7) obj1_6 ; > > and it still fail at the same place: > =E2=80=A6 > Diagnostic Messages for this Task: > java.lang.RuntimeException: > org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:162) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:436= ) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) > at org.apache.hadoop.mapred.Child$4.run(Child.java:255) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:416) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation= .java:1093) > at org.apache.hadoop.mapred.Child.main(Child.java:249) > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOpera= tor.java:198) > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(= MapJoinOperator.java:212) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.= java:1377) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.= java:1381) > at > org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.= java:1381) > at > org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:611) > at > org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) > ... 8 more > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOpera= tor.java:186) > ... 14 more > > -- > wzc1989@gmail.com > =E5=B7=B2=E4=BD=BF=E7=94=A8 Sparrow > > =E5=B7=B2=E4=BD=BF=E7=94=A8 Sparrow > > =E5=9C=A8 2013=E5=B9=B47=E6=9C=8828=E6=97=A5=E6=98=9F=E6=9C=9F=E6=97=A5= =EF=BC=8C=E4=B8=8B=E5=8D=888:08=EF=BC=8Cwzc1989@gmail.com =E5=86=99=E9=81= =93=EF=BC=9A > > hi all: > > We are currently testing hive 0.11 against our production environment and > run into some problems. Some of them are related to the param > "hive.auto.convert.join". > We disable this param and some failed testcases passed. By searching in > hive jira issues I find that the patch in HIVE-4650( > https://issues.apache.org/jira/browse/HIVE-4650) may be helpful. > I compile the newest code in trunk and try the failed testcase in > HIVE-4650, but it doesn't pass. It seems that this issue is not fixed > while it's closed. > > Am I missed something? > > -- > wzc1989@gmail.com > =E5=B7=B2=E4=BD=BF=E7=94=A8 Sparrow > > =E5=B7=B2=E4=BD=BF=E7=94=A8 Sparrow > > > --001a11c2e328bdc8b604e29f7d42 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

Can you also post the output of EXP= LAIN? The execution plan may be helpful to locate the problem.
Thanks,

Yin


On Sun, Jul 28, 2013 at 8:06 PM, <wzc1= 989@gmail.com> wrote:
What I mean by "not pa= ss the testcase in HIVE-4650" is that I compile the trunk code and run= the query in HIVE-4650:
SELECT *
FROM
=C2=A0 (S= ELECT VAL001 x1,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 VAL002 x2,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 VAL003 x3,
=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 VAL004 x4,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 VAL005 y
=C2=A0 =C2=A0FROM
=C2=A0 =C2=A0 =C2=A0(SELECT /*+ mapjoin(v2) */ (VAL001- mu1) * 1/(sd1)= VAL001,(VAL002- mu2) * 1/(sd2) VAL002,(VAL003- mu3) * 1/(sd3) VAL003,(VAL0= 04- mu4) * 1/(sd4) VAL004,(VAL005- mu5) * 1/(sd5) VAL005
=C2=A0 = =C2=A0 =C2=A0 FROM
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (SELECT *
=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0FROM
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(SELECT x1 V= AL001,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0x2 VAL002,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0x3 VAL003,
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0x4 VAL004,
=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0y VAL005
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 FROM cmnt) obj1_3) v3
<= div>=C2=A0 =C2=A0 =C2=A0 JOIN
=C2=A0 =C2=A0 =C2=A0 =C2=A0 (SELECT= count(*) c,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 avg(VAL001) mu1,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 avg(VAL002) mu2,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 avg(VAL003) mu3,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 avg(VAL004) mu= 4,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 avg(VA= L005) mu5,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 stddev_pop(VAL001) sd1,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 stddev_pop(VAL002) sd2,
=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stddev_pop(VAL003) sd3,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stddev_pop(VAL= 004) sd4,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= stddev_pop(VAL005) sd5
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0FROM
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(SELECT *
=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 FROM
=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 (SELECT x1 VAL001,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 x2 VAL002,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 x3 VAL003,
=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 x4 VAL004,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 y VAL005
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0FROM cmnt) obj1_3) v1) v2) obj1_7) obj1_6 ;

and it still fail at t= he same place:
=E2=80=A6=
Diagnostic Messages for this Task:
java.lang.Runt= imeException: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.N= ullPointerException
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.exec.ExecMapp= er.map(ExecMapper.java:162)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.ap= ache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
=C2=A0 =C2=A0= =C2=A0 =C2=A0 at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.jav= a:436)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.mapred.MapTask.run(Ma= pTask.java:372)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.= mapred.Child$4.run(Child.java:255)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at= java.security.AccessController.doPrivileged(Native Method)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at javax.security.auth.Subject.doAs(Subjec= t.java:416)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.secu= rity.UserGroupInformation.doAs(UserGroupInformation.java:1093)
= =C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.mapred.Child.main(Child.ja= va:249)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang= .NullPointerException
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.h= adoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:198)<= /div>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.exec.Map= JoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:212)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.exec.Operator= .cleanUpInputFileChanged(Operator.java:1377)
=C2=A0 =C2=A0 =C2=A0= =C2=A0 at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(= Operator.java:1381)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.had= oop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1381)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.exec.MapOpera= tor.process(MapOperator.java:611)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at = org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 ... 8 more
Caused by: java.lang.Nul= lPointerException
=C2=A0 =C2=A0 =C2=A0 =C2=A0 at org.apache.hadoop.hive.ql.exec.MapJoinO= perator.loadHashTable(MapJoinOperator.java:186)
=C2=A0 =C2=A0 =C2= =A0 =C2=A0 ... 14 more

--=C2=A0
=E5= =B7=B2=E4=BD=BF=E7=94=A8 Sparrow

=E5=B7=B2=E4=BD=BF=E7=94=A8 Sparrow
=20

=E5=9C=A8 2013=E5=B9=B47= =E6=9C=8828=E6=97=A5=E6=98=9F=E6=9C=9F=E6=97=A5=EF=BC=8C=E4=B8=8B=E5=8D=888= :08=EF=BC=8Cwzc1989@= gmail.com =E5=86=99=E9=81=93=EF=BC=9A

hi all:

We are currently testing hive 0.11 against our = production environment and run into some problems. Some of them are related= to the param "hive.auto.convert.join".
We disable this param and some failed=C2=A0= testcases passed. By searching in hive=C2=A0jira issues I find that the pat= ch in HIVE-4650(https://issues.apache.org/jira/browse/HIVE-4650) may be helpful.
I compile the newest code in trunk and try = the failed=C2=A0testcase in HIVE-4650, but it doesn't pass.=C2=A0It seems that this issue= is not fixed while it's closed.

Am I missed something?

--=C2=A0
=E5=B7=B2=E4=BD=BF=E7=94=A8 Sparrow

=E5=B7=B2=E4=BD= =BF=E7=94=A8 Sparrow
=20 =20 =20 =20
=20


--001a11c2e328bdc8b604e29f7d42--