spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fengdong Yu <fengdo...@everstring.com>
Subject Re: There is any way to write from spark to HBase CDH4?
Date Tue, 27 Oct 2015 10:35:29 GMT
Does this released with Spark1.*? or still kept in the trunk?




> On Oct 27, 2015, at 6:22 PM, Adrian Tanase <atanase@adobe.com> wrote:
> 
> Also I just remembered about cloudera’s contribution
> http://blog.cloudera.com/blog/2015/08/apache-spark-comes-to-apache-hbase-with-hbase-spark-module/
<http://blog.cloudera.com/blog/2015/08/apache-spark-comes-to-apache-hbase-with-hbase-spark-module/>
> 
> From: Deng Ching-Mallete
> Date: Tuesday, October 27, 2015 at 12:03 PM
> To: avivb
> Cc: user
> Subject: Re: There is any way to write from spark to HBase CDH4?
> 
> Hi,
> 
> We are using phoenix-spark (http://phoenix.apache.org/phoenix_spark.html <http://phoenix.apache.org/phoenix_spark.html>)
to write data to HBase, but it requires spark 1.3.1+ and phoenix 4.4+. Previously, when we
were still on spark 1.2, we used the HBase API to write directly to HBase.
> 
> For HBase 0.98, it's something like this:
> 
> rdd.foreachPartition(partition => {
>    // create hbase config
>    val hConf = HBaseConfiguration.create()
>    val hTable = new HTable(hConf, "TABLE_1")
>    hTable.setAutoFlush(false)
> 
>    partition.foreach(r => {
>      // generate row key
>      // create row
>      val hRow = new Put(rowKey)
> 
>      // add columns 
>      hRow.add(..)
>     
>      hTable.put(hRow)
>    })
>    hTable.flushCommits()
>    hTable.close()
> })
> 
> HTH,
> Deng
> 
> On Tue, Oct 27, 2015 at 5:36 PM, avivb <aviv@taykey.com <mailto:aviv@taykey.com>>
wrote:
>> I have already try it with https://github.com/unicredit/hbase-rdd <https://github.com/unicredit/hbase-rdd>
and
>> https://github.com/nerdammer/spark-hbase-connector <https://github.com/nerdammer/spark-hbase-connector>
and in both cases I get
>> timeout.
>> 
>> So I would like to know about other option to write from Spark to HBase
>> CDH4.
>> 
>> Thanks!
>> 
>> 
>> 
>> --
>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/There-is-any-way-to-write-from-spark-to-HBase-CDH4-tp25209.html
<http://apache-spark-user-list.1001560.n3.nabble.com/There-is-any-way-to-write-from-spark-to-HBase-CDH4-tp25209.html>
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> 
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org <mailto:user-unsubscribe@spark.apache.org>
>> For additional commands, e-mail: user-help@spark.apache.org
>>  <mailto:user-help@spark.apache.org>


Mime
View raw message