Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DF83B18A98 for ; Fri, 31 Jul 2015 07:17:37 +0000 (UTC) Received: (qmail 45760 invoked by uid 500); 31 Jul 2015 07:17:36 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 45686 invoked by uid 500); 31 Jul 2015 07:17:36 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 45676 invoked by uid 99); 31 Jul 2015 07:17:36 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 31 Jul 2015 07:17:36 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id AF635D9231 for ; Fri, 31 Jul 2015 07:17:35 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.129 X-Spam-Level: *** X-Spam-Status: No, score=3.129 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id tvkr7i_CCTVr for ; Fri, 31 Jul 2015 07:17:34 +0000 (UTC) Received: from mail-ig0-f179.google.com (mail-ig0-f179.google.com [209.85.213.179]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id CA03221231 for ; Fri, 31 Jul 2015 07:17:33 +0000 (UTC) Received: by igr7 with SMTP id 7so11157958igr.0 for ; Fri, 31 Jul 2015 00:17:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=Ue/ySPvV4OAbnGQNZUcm+JdqLLBlewmj76GHKZG6yMg=; b=u/NeuqzxO4LUWn4QGmWow6sEalBEuUM9jVAMQTZvigLzpW8bbmV4DOr+0yXGurSfOd NTivSIQNQW3ZfDizaf0Rc3gPpBBdyWvTCwW3sAfr1GgNiAREDcv7G7TMXli5zHI9JbyZ 076kR7nTXHl8bYMB0znjkOkgKkMX+rL/Ahc3W+jbd44gy0SrwHKI1iUMR9xjagYfCu6I 1XRd+mCpK1nkjAdYLCaYL/JJb7cRobUDv3PWvXuC4faBsxtLtp2GugrPLHoKFSlNBZuQ Jqo3EAaSheNNkwMin+B+5Nttp36ruvxiIpgla+ihr1kYwPe4u+cY5wztCBvP8wPRrEzv 7PIw== MIME-Version: 1.0 X-Received: by 10.50.79.229 with SMTP id m5mr3207041igx.77.1438327052733; Fri, 31 Jul 2015 00:17:32 -0700 (PDT) Received: by 10.64.116.194 with HTTP; Fri, 31 Jul 2015 00:17:32 -0700 (PDT) In-Reply-To: References: Date: Fri, 31 Jul 2015 12:47:32 +0530 Message-ID: Subject: Re: [blocker] ArrayIndexoutofbound in a hive query From: Nitin Pawar To: "user@hive.apache.org" Content-Type: multipart/alternative; boundary=089e0122a3fa0c93c4051c269d2a --089e0122a3fa0c93c4051c269d2a Content-Type: text/plain; charset=UTF-8 is there a different output format or the output table bucketed? can you try putting a not null condition on join columns On Fri, Jul 31, 2015 at 12:45 PM, ravi teja wrote: > Hi Nithin, > Thanks for replying. > The select query runs like a charm, but only when insertion into a table, > then this problem occurs. > > Please find the answers inline. > > > Thanks, > Ravi > > On Fri, Jul 31, 2015 at 12:34 PM, Nitin Pawar > wrote: > >> sorry but i could not find following info >> 1) are you using tez as execution engine? if yes make sure its not >> snapshot version *NO* >> 2) are you using orc file format? if yes then set flag to ignore corrupt >> data *NO, Its Text file format* >> 3) are there nulls on your join condition columns *Yes, there might be >> some* >> if possible share the query and underlying file formats with some sample >> data *I cant really share the query.* >> >> On Fri, Jul 31, 2015 at 12:14 PM, ravi teja wrote: >> >>> Hi, >>> >>> We are facing issue with our hive query with ArrayIndexoutofbound >>> exception. >>> I have tried googling out and I see many users facing the same error, >>> but no solution yet. This is a blocker for our production and we really >>> need help on this. >>> >>> We are using Hive version : 1.3.0. >>> >>> Our query is doing multiple joins(right and left). >>> >>> >>> *Diagnostic Messages for this Task:* >>> Error: java.lang.RuntimeException: >>> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while >>> processing row >>> {"_col0":48436215,"_col1":87269315,"_col2":"\u0000","_col3":"Customer","_col4":null,"_col5":null,"_col6":"CSS >>> Email","_col7":"","_col8":null,"_col9":null,"_col10":null,"_col11":null,"_col12":null,"_col13":null} >>> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172) >>> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54) >>> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450) >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343) >>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:415) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) >>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) >>> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive >>> Runtime Error while processing row >>> {"_col0":48436215,"_col1":87269315,"_col2":"\u0000","_col3":"Customer","_col4":null,"_col5":null,"_col6":"CSS >>> Email","_col7":"","_col8":null,"_col9":null,"_col10":null,"_col11":null,"_col12":null,"_col13":null} >>> at >>> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518) >>> at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163) >>> ... 8 more >>> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: >>> java.lang.ArrayIndexOutOfBoundsException >>> at >>> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:403) >>> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837) >>> at >>> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97) >>> at >>> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162) >>> at >>> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508) >>> ... 9 more >>> Caused by: java.lang.ArrayIndexOutOfBoundsException >>> at java.lang.System.arraycopy(Native Method) >>> at org.apache.hadoop.io.Text.set(Text.java:225) >>> at >>> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryString.init(LazyBinaryString.java:48) >>> at >>> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.uncheckedGetField(LazyBinaryStruct.java:267) >>> at >>> org.apache.hadoop.hive.serde2.lazybinary.LazyBinaryStruct.getField(LazyBinaryStruct.java:204) >>> at >>> org.apache.hadoop.hive.serde2.lazybinary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(LazyBinaryStructObjectInspector.java:64) >>> at >>> org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluator._evaluate(ExprNodeColumnEvaluator.java:94) >>> at >>> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77) >>> at >>> org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65) >>> at >>> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.makeValueWritable(ReduceSinkOperator.java:558) >>> at >>> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:383) >>> ... 13 more >>> >>> >>> FAILED: Execution Error, return code 2 from >>> org.apache.hadoop.hive.ql.exec.mr.MapRedTask >>> >>> >>> >>> Thanks, >>> Ravi >>> >>> >> >> >> -- >> Nitin Pawar >> > > -- Nitin Pawar --089e0122a3fa0c93c4051c269d2a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
is there a different output format or the output tabl= e bucketed?
can you try putting a not null condition on join colu= mns

On = Fri, Jul 31, 2015 at 12:45 PM, ravi teja <raviorteja@gmail.com><= /span> wrote:
Hi Nithin,=
Thanks for replying.
The select query runs like =C2=A0a char= m, but only when insertion into a table, then this problem occurs.

Please find the answers inline.

<= br>
Thanks,
Ravi

On Fri, Jul 31, 2015 at 12:34 PM,= Nitin Pawar <nitinpawar432@gmail.com> wrote:
sorry but i could not find following info
1) are yo= u using tez as execution engine? if yes make sure its not snapshot version = =C2=A0NO
2) are you using or= c file format? if yes then set flag to ignore corrupt data =C2=A0NO, Its Text file format
3) are t= here nulls on your join condition columns =C2=A0= Yes, there might be some
if possible share the query an= d underlying file formats with some sample data =C2=A0 I cant really share the query.

On Fri, Jul 31, 2015 at 12:14 PM, ravi teja <raviorteja@gmail.com= > wrote:
Hi,

We are facing= issue with our hive query with ArrayIndexoutofbound exception.
I= have tried googling out and I see many users facing the same error, but no= solution yet. This is a blocker for our production and we really need help= on this.

We are using Hive version : 1.3.0.

Our query is doing multiple joins(right and left).


Diagnostic Messages for this Ta= sk:
Error: java.lang.RuntimeException: org.apache.hadoop.hive= .ql.metadata.HiveException: Hive Runtime Error while processing row {"= _col0":48436215,"_col1":87269315,"_col2":"\u0= 000","_col3":"Customer","_col4":null,&qu= ot;_col5":null,"_col6":"CSS Email","_col7&quo= t;:"","_col8":null,"_col9":null,"_col10&= quot;:null,"_col11":null,"_col12":null,"_col13&quo= t;:null}
at org.apac= he.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
at org.apache.hadoop.mapred.MapRu= nner.run(MapRunner.java:54)
= at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)<= /div>
at org.apache.hadoop= .mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.jav= a:163)
at java.secur= ity.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.ja= va:415)
at org.apach= e.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)=
at org.apache.hadoo= p.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache= .hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing= row {"_col0":48436215,"_col1":87269315,"_col2&quo= t;:"\u0000","_col3":"Customer","_col4&qu= ot;:null,"_col5":null,"_col6":"CSS Email",&qu= ot;_col7":"","_col8":null,"_col9":null,&= quot;_col10":null,"_col11":null,"_col12":null,&quo= t;_col13":null}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518= )
at org.apache.hado= op.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
... 8 more
Caused by: org.apac= he.hadoop.hive.ql.metadata.HiveException: java.lang.ArrayIndexOutOfBoundsEx= ception
at org.apach= e.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:40= 3)
at org.apache.had= oop.hive.ql.exec.Operator.forward(Operator.java:837)
at org.apache.hadoop.hive.ql.exec.TableSc= anOperator.process(TableScanOperator.java:97)
at org.apache.hadoop.hive.ql.exec.MapOperator$Map= OpCtx.forward(MapOperator.java:162)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOp= erator.java:508)
...= 9 more
Caused by: java.lang.ArrayIndexOutOfBoundsException
=
at java.lang.System.array= copy(Native Method)
= at org.apache.hadoop.io.Text.set(Text.java:225)
at org.apache.hadoop.hive.serde2.lazybinary.Laz= yBinaryString.init(LazyBinaryString.java:48)
at org.apache.hadoop.hive.serde2.lazybinary.LazyBi= naryStruct.uncheckedGetField(LazyBinaryStruct.java:267)
at org.apache.hadoop.hive.serde2.lazybi= nary.LazyBinaryStruct.getField(LazyBinaryStruct.java:204)
at org.apache.hadoop.hive.serde2.lazy= binary.objectinspector.LazyBinaryStructObjectInspector.getStructFieldData(L= azyBinaryStructObjectInspector.java:64)
at org.apache.hadoop.hive.ql.exec.ExprNodeColumnEvaluat= or._evaluate(ExprNodeColumnEvaluator.java:94)
at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluat= or.evaluate(ExprNodeEvaluator.java:77)
at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.eval= uate(ExprNodeEvaluator.java:65)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.makeValueW= ritable(ReduceSinkOperator.java:558)
at org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.proce= ss(ReduceSinkOperator.java:383)
... 13 more


FAILED: Exe= cution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTa= sk



Thanks,
Ravi




--
Nitin Pawar




--
Nitin Pawar
--089e0122a3fa0c93c4051c269d2a--