spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Deng Ching-Mallete <och...@apache.org>
Subject Re: There is any way to write from spark to HBase CDH4?
Date Tue, 27 Oct 2015 10:03:21 GMT
Hi,

We are using phoenix-spark (http://phoenix.apache.org/phoenix_spark.html)
to write data to HBase, but it requires spark 1.3.1+ and phoenix 4.4+.
Previously, when we were still on spark 1.2, we used the HBase API to write
directly to HBase.

For HBase 0.98, it's something like this:

rdd.foreachPartition(partition => {
   // create hbase config
   val hConf = HBaseConfiguration.create()
   val hTable = new HTable(hConf, "TABLE_1")
   hTable.setAutoFlush(false)

   partition.foreach(r => {
     // generate row key
     // create row
     val hRow = new Put(rowKey)

     // add columns
     hRow.add(..)

     hTable.put(hRow)
   })
   hTable.flushCommits()
   hTable.close()
})

HTH,
Deng

On Tue, Oct 27, 2015 at 5:36 PM, avivb <aviv@taykey.com> wrote:

> I have already try it with https://github.com/unicredit/hbase-rdd and
> https://github.com/nerdammer/spark-hbase-connector and in both cases I get
> timeout.
>
> So I would like to know about other option to write from Spark to HBase
> CDH4.
>
> Thanks!
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/There-is-any-way-to-write-from-spark-to-HBase-CDH4-tp25209.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>

Mime
View raw message