spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "sydt (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-19661) Spark-2.1.0 can not connect hbase
Date Wed, 22 Feb 2017 04:18:44 GMT

    [ https://issues.apache.org/jira/browse/SPARK-19661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15877441#comment-15877441
] 

sydt edited comment on SPARK-19661 at 2/22/17 4:18 AM:
-------------------------------------------------------

Yeah,you are right.It is accomplished by hive-hbase-handler-0.13.1.jar actually. In hive,
we can create table by
SET hbase.zookeeper.quorum=zkNode1,zkNode2,zkNode3; 
SET zookeeper.znode.parent=/hbase;
ADD jar /usr/local/apache-hive-0.13.1-bin/lib/hive-hbase-handler-0.13.1.jar;
CREATE EXTERNAL TABLE lxw1234 (
rowkey string,
f1 map<STRING,STRING>,
f2 map<STRING,STRING>,
f3 map<STRING,STRING>
) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,f1:,f2:,f3:")
TBLPROPERTIES ("hbase.table.name" = "lxw1234");
After  
INSERT INTO TABLE lxw1234 
SELECT 'row1' AS rowkey,
map('c3','name3') AS f1,
map('c3','age3') AS f2,
map('c4','job3') AS f3 
FROM DUAL 
limit 1;
now, hive can access table lxw1234 like ordinary external table and it it stored in hbase.
Sparksql has the same syntax parser as hive and it should support this SQL syntax in spark-1.6.2.However,
in spark-2.1.0,it does not. 


was (Author: wangchao2017):
Yeah,you are right.It is accomplished by hive-hbase-handler-0.13.1.jar actually. In hive,
we can create table by
SET hbase.zookeeper.quorum=zkNode1,zkNode2,zkNode3; 
SET zookeeper.znode.parent=/hbase;
ADD jar /usr/local/apache-hive-0.13.1-bin/lib/hive-hbase-handler-0.13.1.jar;
CREATE EXTERNAL TABLE lxw1234 (
rowkey string,
f1 map<STRING,STRING>,
f2 map<STRING,STRING>,
f3 map<STRING,STRING>
) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'
WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,f1:,f2:,f3:")
TBLPROPERTIES ("hbase.table.name" = "lxw1234");
After  
INSERT INTO TABLE lxw1234 
SELECT 'row1' AS rowkey,
map('c3','name3') AS f1,
map('c3','age3') AS f2,
map('c4','job3') AS f3 
FROM DUAL 
limit 1;
now, hive can access table lxw1234 like ordinary external table and it it stored in hbase.


> Spark-2.1.0 can not connect hbase
> ---------------------------------
>
>                 Key: SPARK-19661
>                 URL: https://issues.apache.org/jira/browse/SPARK-19661
>             Project: Spark
>          Issue Type: Bug
>          Components: Deploy
>    Affects Versions: 2.1.0
>            Reporter: sydt
>
> When spark-sql of spark-2.1.0 connect hbase by
> CREATE EXTERNAL TABLE lxw123(  
> rowkey string,  
> f1 map<STRING,STRING>,  
> f2 map<STRING,STRING>,  
> f3 map<STRING,STRING>  
> ) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler'  
> WITH SERDEPROPERTIES ("hbase.columns.mapping" = ":key,f1:,f2:,f3:")  
> TBLPROPERTIES ("hbase.table.name" = "lxw1234");
> it has no response and show 
> Error in query: 
> Operation not allowed: STORED BY(line 6, pos 2)



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message