hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chengxiang Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-10073) Runtime exception when querying HBase with Spark [Spark Branch]
Date Fri, 27 Mar 2015 01:25:54 GMT

    [ https://issues.apache.org/jira/browse/HIVE-10073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14383152#comment-14383152
] 

Chengxiang Li commented on HIVE-10073:
--------------------------------------

[~xuefuz], the root cause should be just like Jimmy mentioned, some hbase table properties
are set to JobConf during checkOutputSpecs, and this method is not invoked in HoS. Actually
Spark checkout output specs while user build RDD graph with certain actions, like PairRDDFunctions::saveAsHadoopDataset,
PairRDDFunctions::saveAsNewAPIHadoopDataset, in HoS, we use foreach as action, and write data
to hadoop storage inside Hive, so it should be Hive's reponsbility to check output specs.

> Runtime exception when querying HBase with Spark [Spark Branch]
> ---------------------------------------------------------------
>
>                 Key: HIVE-10073
>                 URL: https://issues.apache.org/jira/browse/HIVE-10073
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: spark-branch
>            Reporter: Jimmy Xiang
>            Assignee: Jimmy Xiang
>             Fix For: spark-branch
>
>         Attachments: HIVE-10073.1-spark.patch, HIVE-10073.2-spark.patch, HIVE-10073.3-spark.patch
>
>
> When querying HBase with Spark, we got 
> {noformat}
>  Caused by: java.lang.IllegalArgumentException: Must specify table name
> at org.apache.hadoop.hbase.mapreduce.TableOutputFormat.setConf(TableOutputFormat.java:188)
> at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
> at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
> at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveOutputFormat(HiveFileFormatUtils.java:276)
> at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveOutputFormat(HiveFileFormatUtils.java:266)
> at org.apache.hadoop.hive.ql.exec.FileSinkOperator.initializeOp(FileSinkOperator.java:331)
> {noformat}
> But it works fine for MapReduce.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message