hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-13992) Integrate SparkOnHBase into HBase
Date Thu, 09 Jul 2015 18:58:04 GMT

    [ https://issues.apache.org/jira/browse/HBASE-13992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621073#comment-14621073

Sean Busbey commented on HBASE-13992:

Notes from call today where [~apurtell] and I helped walk through a plan for this with [~ted.m].

h3. Overall Roadmap

* start with current capabilities refactored into a module with hbase packages
* initial support targets Spark 1.3
* later add RDD extension to expose hbase-specific functionality
* later add data frame support, likely as additional module
* start with master-only,  use backport patch for branch-1 post refactor and docs 

h3. Minimums for Landing on Master

* use an isolated top-level module to reduce impact on rest of project, modeled after the
phoenix-spark module
** hbase-spark
** should help limit reach of dependency conflicts. look to phoenix-spark for any conflict
* target spark 1.3 as upstream dependency
** already works with 1.3
** 1.3 seen as able to reach a large existing user base and stable
** gap: data frames are missing key functionality pre-1.4.
** gap: unknown spark compatibility promises, history of scala version changes may limit how
many future versions this can work with.
* move existing implementation classes into an hbase subpackage
**  org.apache.hadoop.hbase.spark
** there is currently a workaround to access a spark-package-private set of configs, needed
for pre-1.3. since 1.3 is our minimum expected version, remove this workaround.
* leave examples as a part of the single module definition for first landing
* leave tests, including scala tests, as unit tests on the single module for first landing

h3. Follow on for branch-1

* documentation - need to add a ref guide section on using the spark bindings
* examples refactoring - examples should be moved out of the primary module into something
like hbase-spark-examples
* tests refactoring - java tests should be categorized to match rest of project. Scala tests
should be moved to an IT

> Integrate SparkOnHBase into HBase
> ---------------------------------
>                 Key: HBASE-13992
>                 URL: https://issues.apache.org/jira/browse/HBASE-13992
>             Project: HBase
>          Issue Type: Bug
>            Reporter: Ted Malaska
>            Assignee: Ted Malaska
>              Labels: spark
>             Fix For: 2.0.0
> This Jira is to ask if SparkOnHBase can find a home in side HBase core.
> Here is the github: 
> https://github.com/cloudera-labs/SparkOnHBase
> I am the core author of this project and the license is Apache 2.0
> A blog explaining this project is here
> http://blog.cloudera.com/blog/2014/12/new-in-cloudera-labs-sparkonhbase/
> A spark Streaming example is here
> http://blog.cloudera.com/blog/2014/11/how-to-do-near-real-time-sessionization-with-spark-streaming-and-apache-hadoop/
> A real customer using this in produce is blogged here
> http://blog.cloudera.com/blog/2015/03/how-edmunds-com-used-spark-streaming-to-build-a-near-real-time-dashboard/
> Please debate and let me know what I can do to make this happen.

This message was sent by Atlassian JIRA

View raw message