spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <>
Subject Re: State of the Build
Date Sat, 07 Nov 2015 00:43:02 GMT
bq. include an sbt jar in the source repo

Can you clarify which sbt jar (by path) ?

I tried 'git log' on the following files but didn't see commit history:


On Fri, Nov 6, 2015 at 4:25 PM, Jakob Odersky <> wrote:

> [Reposting to the list again, I really should double-check that
> reply-to-all button]
> in the mean-time, as a light Friday-afternoon patch I was thinking about
> splitting the ~600loc-single-build sbt file into something more manageable
> like the Akka build (without changing any dependencies or settings). I know
> its pretty trivial and not very important, but it might make things easier
> to add/refactor in the future.
> Also, why do we include an sbt jar in the source repo, especially if it is
> used only as an internal development tool?
> On 6 November 2015 at 15:29, Patrick Wendell <> wrote:
>> I think there are a few minor differences in the dependency graph that
>> arise from this. For a given user, the probability it affects them is low -
>> it needs to conflict with a library a user application is using. However
>> the probability it affects *some users* is very high and we do see small
>> changes crop up fairly frequently.
>> My feeling is mostly pragmatic... if we can get things working to
>> standardize on Maven-style resolution by upgrading SBT, let's do it. If
>> that's not tenable, we can evaluate alternatives.
>> - Patrick
>> On Fri, Nov 6, 2015 at 3:07 PM, Marcelo Vanzin <>
>> wrote:
>>> On Fri, Nov 6, 2015 at 3:04 PM, Koert Kuipers <> wrote:
>>> > if i understand it correctly it would cause compatibility breaks for
>>> > applications on top of spark, because those applications use the exact
>>> same
>>> > current resolution logic (so basically they are maven apps), and the
>>> change
>>> > would make them inconsistent?
>>> I think Patrick said it could cause compatibility breaks because
>>> switching to sbt's version resolution means Spark's dependency tree
>>> would change. Just to cite the recent example, you'd get Guava 16
>>> instead of 14 (let's ignore that Guava is currently mostly shaded in
>>> Spark), so if your app depended transitively on Guava and used APIs
>>> from 14 that are not on 16, it would break.
>>> --
>>> Marcelo

View raw message