Return-Path: Delivered-To: apmail-hadoop-hive-user-archive@locus.apache.org Received: (qmail 25981 invoked from network); 2 Dec 2008 09:45:19 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2) by minotaur.apache.org with SMTP; 2 Dec 2008 09:45:19 -0000 Received: (qmail 38469 invoked by uid 500); 2 Dec 2008 09:45:31 -0000 Delivered-To: apmail-hadoop-hive-user-archive@hadoop.apache.org Received: (qmail 38447 invoked by uid 500); 2 Dec 2008 09:45:31 -0000 Mailing-List: contact hive-user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hive-user@hadoop.apache.org Delivered-To: mailing list hive-user@hadoop.apache.org Received: (qmail 38438 invoked by uid 99); 2 Dec 2008 09:45:31 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 01:45:31 -0800 X-ASF-Spam-Status: No, hits=-1.8 required=10.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_MED,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of zshao@facebook.com designates 204.15.23.140 as permitted sender) Received: from [204.15.23.140] (HELO mailout-sf2p.facebook.com) (204.15.23.140) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 02 Dec 2008 09:44:04 +0000 Received: from mail.thefacebook.com (sc-hub01.thefacebook.com [192.168.18.104]) by pp01.sf2p.tfbnw.net (8.14.1/8.14.1) with ESMTP id mB29iptr030999 (version=TLSv1/SSLv3 cipher=RC4-MD5 bits=128 verify=NOT) for ; Tue, 2 Dec 2008 01:44:51 -0800 Received: from SC-MBXC1.TheFacebook.com ([192.168.18.100]) by sc-hub01.TheFacebook.com ([192.168.18.104]) with mapi; Tue, 2 Dec 2008 01:44:50 -0800 From: Zheng Shao To: "hive-user@hadoop.apache.org" Date: Tue, 2 Dec 2008 01:44:50 -0800 Subject: RE: Index Expressions Thread-Topic: Index Expressions Thread-Index: AclUYXGYbYiuK7/ZQtSgpt56qefbgwAANALQ Message-ID: References: <6869207D-3B12-424F-BD84-CFD33F372EFC@besquared.net> In-Reply-To: <6869207D-3B12-424F-BD84-CFD33F372EFC@besquared.net> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: multipart/alternative; boundary="_000_A132F89F9B9DF241B1C3C33E7299BEB90466B19E7BSCMBXC1TheFac_" MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=1.12.7400:2.4.4,1.2.40,4.0.166 definitions=2008-12-02_02:2008-11-24,2008-12-02,2008-12-02 signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 ipscore=0 phishscore=0 bulkscore=0 adultscore=0 classifier=spam adjust=0 reason=mlx engine=5.0.0-0805090000 definitions=main-0807290170 X-Virus-Checked: Checked by ClamAV on apache.org --_000_A132F89F9B9DF241B1C3C33E7299BEB90466B19E7BSCMBXC1TheFac_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Hi Josh, This is a known problem. If any of the columns are map or list, you have to specif= y the ITEM terminator and KEY terminator. Please see the complete CREATE ta= ble command syntax. Let me know if it works or not. Zheng From: Josh Ferguson [mailto:josh@besquared.net] Sent: Tuesday, December 02, 2008 1:36 AM To: hive-user@hadoop.apache.org Subject: Index Expressions I'm getting a bunch of errors during my reduce step while doing a join betw= een two string fields for some reason. java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index ex= pression on string at org.apache.hadoop.hive.ql.exec.ExprNodeIndexEvaluator.evaluate= (ExprNodeIndexEvaluator.java:64) at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(= ExprNodeFuncEvaluator.java:72) at org.apache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(= ExprNodeFuncEvaluator.java:72) at org.apache.hadoop.hive.ql.exec.FilterOperator.process(FilterOp= erator.java:67) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:= 262) at org.apache.hadoop.hive.ql.exec.JoinOperator.createForwardJoinO= bject(JoinOperator.java:257) at org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOper= ator.java:477) at org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOper= ator.java:467) at org.apache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOper= ator.java:467) at org.apache.hadoop.hive.ql.exec.JoinOperator.checkAndGenObject(= JoinOperator.java:507) at org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOpera= tor.java:489) at org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.= java:140) at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430) at org.apache.hadoop.mapred.Child.main(Child.java:155) The Query is SELECT activities.*, users.* FROM activities LEFT OUTER JOIN users ON activ= ities.actor_id =3D users.id WHERE activities.dataset=3D'poke' AND activitie= s.properties['verb'] =3D 'Dance'; hive> DESCRIBE activities; OK actor_id string actee_id string properties map account string application string dataset string hour int hive> DESCRIBE users; OK id string properties map account string application string dataset string hour int Thanks Josh --_000_A132F89F9B9DF241B1C3C33E7299BEB90466B19E7BSCMBXC1TheFac_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

Hi Josh,

 

This is a known problem.

 

If any of the columns are map<xxx,xxx> or list<xxx,= xxx>, you have to specify the ITEM terminator and KEY terminator. Please see the = complete CREATE table command syntax.

 

Let me know if it works or not.

 

Zheng

From: Josh Ferguson [mailto:josh@besquared.net]
Sent: Tuesday, December 02, 2008 1:36 AM
To: hive-user@hadoop.apache.org
Subject: Index Expressions

 

I'm getting a bunch of errors during my reduce step wh= ile doing a join between two string fields for some reason.

 

java.lang.RuntimeException: Hive 2 Internal error: cannot evaluate index expression on string<= /p> 

          at=
 org.apache.hadoop.hive.ql.exec.ExprNodeIndexEvaluator.evaluate(ExprNodeInd=
exEvaluator.java:64)
          at org.ap=
ache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluat=
or.java:72)
          at org.ap=
ache.hadoop.hive.ql.exec.ExprNodeFuncEvaluator.evaluate(ExprNodeFuncEvaluat=
or.java:72)
          at org.ap=
ache.hadoop.hive.ql.exec.FilterOperator.process(FilterOperator.java:67)
          at org.ap=
ache.hadoop.hive.ql.exec.Operator.forward(Operator.java:262)
          at org.ap=
ache.hadoop.hive.ql.exec.JoinOperator.createForwardJoinObject(JoinOperator.=
java:257)
          at org.ap=
ache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:477)=

          at org.ap=
ache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:467)=

          at org.ap=
ache.hadoop.hive.ql.exec.JoinOperator.genObject(JoinOperator.java:467)=

          at org.ap=
ache.hadoop.hive.ql.exec.JoinOperator.checkAndGenObject(JoinOperator.java:5=
07)
          at org.ap=
ache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:489)<=
/o:p>
          at org.ap=
ache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:140)
          at org.ap=
ache.hadoop.mapred.ReduceTask.run(ReduceTask.java:430)
          at org.ap=
ache.hadoop.mapred.Child.main(Child.java:155)
 
The Query is
SELECT activities.*, users.* FROM ac=
tivities LEFT OUTER JOIN users ON activities.actor_id =3D users.id WHERE ac=
tivities.dataset=3D'poke' AND activities.properties['verb'] =3D 'Dance';









hive>
DESCRIBE activities;









OK









actor_id        st=
ring









actee_id        st=
ring









properties      map<string,=
string>









account string









application     string









dataset string









hour    int









 









hive>
DESCRIBE users;



OK



id      string



properties      map<string,string>



account string



application     string



dataset string



hour    int









 









Thanks









 









Josh









 
--_000_A132F89F9B9DF241B1C3C33E7299BEB90466B19E7BSCMBXC1TheFac_--