hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wojciech Indyk <wojciechin...@gmail.com>
Subject Re: Spark on Hbase
Date Sun, 20 Mar 2016 17:13:43 GMT
Hello Rachana!
I use HBase with Spark Streaming. My solution is to create a singleton
with HConnection object for each JVM and refer there from
foreachPartition(), creating table connection (it's cheap according to
HBase documentation) for each streaming iteration. I close connections
on SparkShutdownHook.

--
Kind regards/ Pozdrawiam,
Wojciech Indyk
http://datacentric.pl


2016-03-09 7:04 GMT+01:00 Rachana Srivastava
<rachanasrivastav@yahoo.com.invalid>:
> I am trying to integrate SparkStreaming with HBase.  I am calling following APIs to connect
to HBase
>
> HConnection hbaseConnection = HConnectionManager.createConnection(conf);hBaseTable =
hbaseConnection.getTable(hbaseTable);
> Since I cannot get the connection and broadcast the connection each API call to get data
from HBase is very expensive.  I tried using JavaHBaseContext (JavaHBaseContext hbaseContext
= new JavaHBaseContext(jsc, conf)) by using hbase-spark library in CDH 5.5 but I cannot import
the library from maven.  Has anyone been able to successfully resolve this issue.

Mime
View raw message