hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sean Busbey (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-16179) Fix compilation errors when building hbase-spark against Spark 2.0
Date Thu, 10 Aug 2017 04:17:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-16179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121036#comment-16121036

Sean Busbey commented on HBASE-16179:

bq. I am back in US. There should be no ambiguity who would be doing the work for this JIRA.

Good to hear.

Looks like we are going to support Scala 2.10 and 2.11 with Spark 1.6 and 2.x combinations.
How many artifacts would be produced by the build process ?

we support Scala 2.10 and 2.11 with and spark 2.0 to spark 2.2 combinations, no more spark1.6

We don't need to work with Spark 1.6, agreed. Agreed we need to run on Spark 2.z. currently
the [spark dist area|https://dist.apache.org/repos/dist/release/spark/] has a 2.0, 2.1, and
2.2 version. (presuming we get HBase 2.0 out the door before there's a Spark 2.3)

We definitely need a build process that makes appropriate artifacts for runtime of the set
of scala x spark versions. I don't know if we need a different compilation for the different
spark minor versions. Presuming worse case (needing a different artifact for each minor spark
version), that'd be 6 artifacts.

bq.  Maybe for this jira, we do not need to consider artifacts, and open an other jira to
talk about the artifacts? I am not so sure about this part.

I think Ted's already on track to generate multiple artifacts, presuming we need them. That
said, if he'd rather not deal with the Spark 2.z minor version runtime compatibility, I'm
fine with just making the current module work with Spark 2 for this jira so long as we have
some jira that tracks getting everything done for the set of scala x spark versions.

> Fix compilation errors when building hbase-spark against Spark 2.0
> ------------------------------------------------------------------
>                 Key: HBASE-16179
>                 URL: https://issues.apache.org/jira/browse/HBASE-16179
>             Project: HBase
>          Issue Type: Bug
>          Components: spark
>            Reporter: Ted Yu
>            Assignee: Ted Yu
>            Priority: Critical
>              Labels: build
>             Fix For: 2.0.0
>         Attachments: 16179.v0.txt, 16179.v10.txt, 16179.v11.txt, 16179.v12.txt, 16179.v12.txt,
16179.v12.txt, 16179.v13.txt, 16179.v15.txt, 16179.v16.txt, 16179.v18.txt, 16179.v19.txt,
16179.v19.txt, 16179.v1.txt, 16179.v1.txt, 16179.v20.txt, 16179.v22.txt, 16179.v23.txt, 16179.v24.txt,
16179.v25.txt, 16179.v26.txt, 16179.v27.txt, 16179.v4.txt, 16179.v5.txt, 16179.v7.txt, 16179.v8.txt,
> I tried building hbase-spark module against Spark-2.0 snapshot and got the following
compilation errors:
> http://pastebin.com/bg3w247a
> Some Spark classes such as DataTypeParser and Logging are no longer accessible to downstream
> hbase-spark module should not depend on such classes.

This message was sent by Atlassian JIRA

View raw message