spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: State of the Build
Date Sat, 07 Nov 2015 00:43:02 GMT
bq. include an sbt jar in the source repo

Can you clarify which sbt jar (by path) ?

I tried 'git log' on the following files but didn't see commit history:

./build/sbt-launch-0.13.7.jar
./build/zinc-0.3.5.3/lib/sbt-interface.jar
./sbt/sbt-launch-0.13.2.jar
./sbt/sbt-launch-0.13.5.jar

On Fri, Nov 6, 2015 at 4:25 PM, Jakob Odersky <jodersky@gmail.com> wrote:

> [Reposting to the list again, I really should double-check that
> reply-to-all button]
>
> in the mean-time, as a light Friday-afternoon patch I was thinking about
> splitting the ~600loc-single-build sbt file into something more manageable
> like the Akka build (without changing any dependencies or settings). I know
> its pretty trivial and not very important, but it might make things easier
> to add/refactor in the future.
>
> Also, why do we include an sbt jar in the source repo, especially if it is
> used only as an internal development tool?
>
> On 6 November 2015 at 15:29, Patrick Wendell <pwendell@gmail.com> wrote:
>
>> I think there are a few minor differences in the dependency graph that
>> arise from this. For a given user, the probability it affects them is low -
>> it needs to conflict with a library a user application is using. However
>> the probability it affects *some users* is very high and we do see small
>> changes crop up fairly frequently.
>>
>> My feeling is mostly pragmatic... if we can get things working to
>> standardize on Maven-style resolution by upgrading SBT, let's do it. If
>> that's not tenable, we can evaluate alternatives.
>>
>> - Patrick
>>
>> On Fri, Nov 6, 2015 at 3:07 PM, Marcelo Vanzin <vanzin@cloudera.com>
>> wrote:
>>
>>> On Fri, Nov 6, 2015 at 3:04 PM, Koert Kuipers <koert@tresata.com> wrote:
>>> > if i understand it correctly it would cause compatibility breaks for
>>> > applications on top of spark, because those applications use the exact
>>> same
>>> > current resolution logic (so basically they are maven apps), and the
>>> change
>>> > would make them inconsistent?
>>>
>>> I think Patrick said it could cause compatibility breaks because
>>> switching to sbt's version resolution means Spark's dependency tree
>>> would change. Just to cite the recent example, you'd get Guava 16
>>> instead of 14 (let's ignore that Guava is currently mostly shaded in
>>> Spark), so if your app depended transitively on Guava and used APIs
>>> from 14 that are not on 16, it would break.
>>>
>>> --
>>> Marcelo
>>>
>>
>>
>

Mime
View raw message