hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brock Noland (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-6117) mapreduce.RecordReader instance needs to be initialized
Date Mon, 30 Dec 2013 14:44:52 GMT

    [ https://issues.apache.org/jira/browse/HIVE-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13858837#comment-13858837
] 

Brock Noland commented on HIVE-6117:
------------------------------------

Hey guys,

I think I see the issue here. o.a.h.mapreduce.RecordReader has an initialize method which
is called by the mapper. o.a.h.mapred.RecordReader does not have an initialize method. Hive
uses the mapred API so you can see here: https://github.com/apache/hive/blob/fb63a28cd5fddb5e5c974cab84cd9c3a4155e40d/hbase-handler/src/java/org/apache/hadoop/hive/hbase/HiveHBaseTableInputFormat.java#L178

that we wrap the mapreduce RR from HBase in a mapred RR. Therefore it's hive's responsibility
to call initialize which we are not doing. Therefore the patch looks correct and I will commit.

> mapreduce.RecordReader instance needs to be initialized
> -------------------------------------------------------
>
>                 Key: HIVE-6117
>                 URL: https://issues.apache.org/jira/browse/HIVE-6117
>             Project: Hive
>          Issue Type: Bug
>          Components: HBase Handler
>            Reporter: Nick Dimiduk
>            Assignee: Nick Dimiduk
>         Attachments: 6117.00.patch, HIVE-6117.0.patch
>
>
> The HBase storage handler makes use of a mapreduce.RecordReader instance but does not
initialize it when consumed from local context. This results in a NPE for some queries, for
instance
> {noformat}
> create table hbase_1(key string, age int) stored by 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
with serdeproperties ( "hbase.columns.mapping" = "info:age");
> insert overwrite table hbase_1 select name, SUM(age) from studenttab10k group by name;
> select * from hbase_1;
> {noformat}
> The select statement throws the following exception
> {noformat}
> 13/12/18 01:30:32 ERROR CliDriver: Failed with exception java.io.IOException:java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:551)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:489)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1494)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
> at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:446)
> at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:456)
> at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:737)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hbase.mapreduce.TableRecordReaderImpl.nextKeyValue(TableRecordReaderImpl.java:196)
> at org.apache.hadoop.hbase.mapreduce.TableRecordReader.nextKeyValue(TableRecordReader.java:138)
> at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:234)
> at org.apache.hadoop.hive.hbase.HiveHBaseTableInputFormat$1.next(HiveHBaseTableInputFormat.java:193)
> at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:521)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message