beam-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amit Sela (JIRA)" <>
Subject [jira] [Commented] (BEAM-1920) Add Spark 2.x support in Spark runner
Date Sun, 09 Apr 2017 09:08:41 GMT


Amit Sela commented on BEAM-1920:

I'm considering Spark 2.x as default and {{1.6.3}} as the profile-enabled version because
there are many issues with flaky tests now that I believe that should be resolved as a part
of a bunch of bug fixes introduced in Spark {{2.0.0}} and {{2.0.1}}.

The only Con I can think of for that is that Spark 2.x dependency would provide an easy path
to new, incompatible, APIs in Spark so we'd have to think carefully how we test for {{1.6.3}}
breaking changes.  

> Add Spark 2.x support in Spark runner
> -------------------------------------
>                 Key: BEAM-1920
>                 URL:
>             Project: Beam
>          Issue Type: Improvement
>          Components: runner-spark
>            Reporter: Jean-Baptiste Onofré
>            Assignee: Jean-Baptiste Onofré
> I have a branch working with both Spark 1 and Spark 2 backend.
> I'm preparing a pull request about that.

This message was sent by Atlassian JIRA

View raw message