hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhan Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-14801) Enhance the Spark-HBase connector catalog with json format
Date Thu, 10 Mar 2016 19:39:40 GMT

    [ https://issues.apache.org/jira/browse/HBASE-14801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15189831#comment-15189831
] 

Zhan Zhang commented on HBASE-14801:
------------------------------------

[~jerryhe] The schema is defined on the application level. The application can have the catalog
persisted in some storage or defined directly in the code. Following is a short description.

1. table defined the hbase table to operate on.
2. in the columns: the first dimension defined the column name in spark-sql, second dimension(cf)
defined the column family correspondingly in hbase table, and third dimension (col) defined
the column qualifier, and type defines the datatype in sparksql, serdes defined the customized
serdes.
3. rowkey defined the composite key in the hbase key part, separated by ":", correspondingly
the "cf" part has to be defined as "rowkey".

    val catalog = s"""{
                      |"table":{"namespace":"default", "name":"htable"},
                      |"rowkey":"key1:key2",
                      |"columns":{
                      |"col1":{"cf":"rowkey", "col":"key1", "type":"string"},
                      |"col2":{"cf":"rowkey", "col":"key2", "type":"double"},
                      |"col3":{"cf":"cf1", "col":"col2", "type":"binary"},
                      |"col4":{"cf":"cf1", "col":"col3", "type":"timestamp"},
                      |"col5":{"cf":"cf1", "col":"col4", "type":"double", "serdes":"${classOf[DoubleSerDes].getName}"},
                      |"col6":{"cf":"cf1", "col":"col5", "type":"$map"},
                      |"col7":{"cf":"cf1", "col":"col6", "type":"$array"},
                      |"col8":{"cf":"cf1", "col":"col7", "type":"$arrayMap"}
                      |}
                      |}""".stripMargin

> Enhance the Spark-HBase connector catalog with json format
> ----------------------------------------------------------
>
>                 Key: HBASE-14801
>                 URL: https://issues.apache.org/jira/browse/HBASE-14801
>             Project: HBase
>          Issue Type: Sub-task
>            Reporter: Zhan Zhang
>            Assignee: Zhan Zhang
>         Attachments: HBASE-14801-1.patch, HBASE-14801-2.patch, HBASE-14801-3.patch, HBASE-14801-4.patch,
HBASE-14801-5.patch, HBASE-14801-6.patch, HBASE-14801-7.patch, HBASE-14801-8.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message