spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Armbrust <>
Subject Re: mvn or sbt for studying and developing Spark?
Date Mon, 17 Nov 2014 19:47:45 GMT
> * I moved from sbt to maven in June specifically due to Andrew Or's
> describing mvn as the default build tool.  Developers should keep in mind
> that jenkins uses mvn so we need to run mvn before submitting PR's - even
> if sbt were used for day to day dev work

To be clear, I think that the PR builder actually uses sbt
<> currently,
but there are master builds that make sure maven doesn't break (amongst
other things).

> *  In addition, as Sean has alluded to, the Intellij seems to comprehend
> the maven builds a bit more readily than sbt

Yeah, this is a very good point.  I have used `sbt/sbt gen-idea` in the
past, but I'm currently using the maven integration of inteliJ since it
seems more stable.

> * But for command line and day to day dev purposes:  sbt sounds great to
> use  Those sound bites you provided about exposing built-in test databases
> for hive and for displaying available testcases are sweet.  Any
> easy/convenient way to see "more of " those kinds of facilities available
> through sbt ?

The Spark SQL developer readme
<> has a little bit of this,
but we really should have some documentation on using SBT as well.

 Integrating with those systems is generally easier if you are also working
> with Spark in Maven.  (And I wouldn't classify all of those Maven-built
> systems as "legacy", Michael :)

Also a good point, though I've seen some pretty clever uses of sbt's
external project references to link spark into other projects.  I'll
certainly admit I have a bias towards new shiny things in general though,
so my definition of legacy is probably skewed :)

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message