hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vikram Dixit K (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-11216) UDF GenericUDFMapKeys throws NPE when a null map value is passed in
Date Fri, 11 Sep 2015 22:20:45 GMT

    [ https://issues.apache.org/jira/browse/HIVE-11216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741629#comment-14741629
] 

Vikram Dixit K commented on HIVE-11216:
---------------------------------------

Don't we need a unit test for this? I tried creating one but I am not sure how to add the
Null value in the map. Can someone please help?

{code}
create table tab10(id int, dud map<string,string>)
row format delimited
fields terminated by ',';

insert overwrite table tab10
select 1,map(NULL) from src limit 1;

select id, map_keys(dud) from tab10;
{code}

Doesn't work. When I tried creating the map as 

{code}
insert overwrite table tab10
select 1,map() from src limit 1;

select id, map_keys(dud) from tab10;
{code}

I did not see any failure without this patch. Any pointers here? Do I just try a load table?

> UDF GenericUDFMapKeys throws NPE when a null map value is passed in
> -------------------------------------------------------------------
>
>                 Key: HIVE-11216
>                 URL: https://issues.apache.org/jira/browse/HIVE-11216
>             Project: Hive
>          Issue Type: Bug
>          Components: UDF
>    Affects Versions: 1.2.0
>            Reporter: Yibing Shi
>            Assignee: Yibing Shi
>             Fix For: 1.3.0, 2.0.0
>
>         Attachments: HIVE-11216.1.patch, HIVE-11216.patch
>
>
> We can reproduce the problem as below:
> {noformat}
> hive> show create table map_txt;
> OK
> CREATE  TABLE `map_txt`(
>   `id` int,
>   `content` map<int,string>)
> ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> STORED AS INPUTFORMAT
>   'org.apache.hadoop.mapred.TextInputFormat'
> OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
> ...
> Time taken: 0.233 seconds, Fetched: 18 row(s)
> hive> select * from map_txt;
> OK
> 1       NULL
> Time taken: 0.679 seconds, Fetched: 1 row(s)
> hive> select id, map_keys(content) from map_txt;
> ....
> Error during job, obtaining debugging information...
> Examining task ID: task_1435534231122_0025_m_000000 (and more) from job job_1435534231122_0025
> Task with the most failures(4):
> -----
> Task ID:
>   task_1435534231122_0025_m_000000
> URL:
>   http://host-10-17-80-40.coe.cloudera.com:8088/taskdetails.jsp?jobid=job_1435534231122_0025&tipid=task_1435534231122_0025_m_000000
> -----
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
Hive Runtime Error while processing row {"id":1,"content":null}
>         at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:198)
>         at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
>         at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:422)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {"id":1,"content":null}
>         at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:559)
>         at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:180)
>         ... 8 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Error evaluating map_keys(content)
>         at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
>         at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
>         at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:92)
>         at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:796)
>         at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:549)
>         ... 9 more
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
>         at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
>         at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
>         at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
>         at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:79)
>         ... 13 more
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> MapReduce Jobs Launched:
> Stage-Stage-1: Map: 1   HDFS Read: 0 HDFS Write: 0 FAIL
> hive>
> {noformat}
> The error is as below (in mappers):
> {noformat}
> Caused by: java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.udf.generic.GenericUDFMapKeys.evaluate(GenericUDFMapKeys.java:64)
>         at org.apache.hadoop.hive.ql.exec.ExprNodeGenericFuncEvaluator._evaluate(ExprNodeGenericFuncEvaluator.java:166)
>         at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:77)
>         at org.apache.hadoop.hive.ql.exec.ExprNodeEvaluator.evaluate(ExprNodeEvaluator.java:65)
>         at org.apache.hadoop.hive.ql.exec.KeyWrapperFactory$ListKeyWrapper.getNewKey(KeyWrapperFactory.java:113)
>         at org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:778)
>         ... 17 more
> {noformat}
> Looking at the source code:
> {code}
>   public Object evaluate(DeferredObject[] arguments) throws HiveException {
>     retArray.clear();
>     Object mapObj = arguments[0].get();
>     retArray.addAll(mapOI.getMap(mapObj).keySet());
>     return retArray;
>   }
> {code}
> It is obvious that we will have a NPE when a NULL map value is passed in



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message