hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Imran Rashid (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
Date Wed, 28 Sep 2016 19:00:23 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15530546#comment-15530546

Imran Rashid commented on HBASE-16179:

ah, sorry I forgot that you were making JavaHBaseContext.scala work with both versions of
spark.  I'd at least include a comment to that effect.  As is, it chooses the branch at runtime,
as opposed to choosing which one at compile time by having two different implementations which
get chosen by the profile.  (Doesn't really matter to me which one you do, just noting the
difference.)  A minor nit, if you are going to stick w/ doing it at runtime, I'd write it
with pattern matching (this may also fix the compile errors w/ 1.6):

def fn = (it: Iterator[T], conn: Connection) => {
    // the return type is different in spark 1.x & 2.x, we handle both cases
    f.call(asJavaIterator(it), conn) match {
      // spark 1.x
      case iterable: Iterable[R] => iterable.iterator()
      // spark 2.x
      case iterator: Iterator[R] => iterator

on HBaseTableScanRDD -- I am surprised it can use the logging methods from spark's internal
Logging, thought that was against scala's scoping rules.  But I guess I was wrong if it works.

> Fix compilation errors when building hbase-spark against Spark 2.0
> ------------------------------------------------------------------
>                 Key: HBASE-16179
>                 URL: https://issues.apache.org/jira/browse/HBASE-16179
>             Project: HBase
>          Issue Type: Bug
>          Components: spark
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>             Fix For: 2.0.0
>         Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v4.txt, 16179.v5.txt,
16179.v7.txt, 16179.v8.txt, 16179.v9.txt
> I tried building hbase-spark module against Spark-2.0 snapshot and got the following
compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer accessible to downstream
> hbase-spark module should not depend on such classes.

This message was sent by Atlassian JIRA

View raw message