hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mike Drob (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
Date Mon, 05 Mar 2018 23:05:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16386922#comment-16386922

Mike Drob commented on HBASE-16179:

I've been doing some reading on this for you, found 

the last comment on SI-8813 provides a good clue

Looking at NewHadoopRDD in the spark core codebase, I found it looks like this:
class NewHadoopRDD[K, V](
    sc : SparkContext,
    inputFormatClass: Class[_ <: InputFormat[K, V]],
    keyClass: Class[K],
    valueClass: Class[V],
    @transient private val _conf: Configuration)
  extends RDD[(K, V)](sc, Nil) with Logging {

I'm not sure if we need to back off the @transient annotation on some of our own fields then
or if we need to explicitly declare them as {{private val}} now. Can you experiment with it
and let us know what works? It would be a good idea to run the IntegrationTestSparkBulkLoad
as a sanity check too.

> Fix compilation errors when building hbase-spark against Spark 2.0
> ------------------------------------------------------------------
>                 Key: HBASE-16179
>                 URL: https://issues.apache.org/jira/browse/HBASE-16179
>             Project: HBase
>          Issue Type: Bug
>          Components: spark
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>            Priority: Critical
>              Labels: build
>             Fix For: 3.0.0
>         Attachments: 16179.v0.txt, 16179.v1.txt, 16179.v1.txt, 16179.v10.txt, 16179.v11.txt,
16179.v12.txt, 16179.v12.txt, 16179.v12.txt, 16179.v13.txt, 16179.v15.txt, 16179.v16.txt,
16179.v18.txt, 16179.v19.txt, 16179.v19.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt,
16179.v24.txt, 16179.v25.txt, 16179.v26.txt, 16179.v27.txt, 16179.v28.txt, 16179.v28.txt,
16179.v29.txt, 16179.v30.txt, 16179.v31.txt, 16179.v32.txt, 16179.v33.txt, 16179.v34.txt,
16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt, 16179.v9.txt, HBASE-16179.v29.patch
> I tried building hbase-spark module against Spark-2.0 snapshot and got the following
compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer accessible to downstream
> hbase-spark module should not depend on such classes.

This message was sent by Atlassian JIRA

View raw message