spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ryan Williams <>
Subject Re: Spurious test failures, testing best practices
Date Tue, 02 Dec 2014 23:39:40 GMT
Marcelo: by my count, there are 19 maven modules in the codebase. I am
typically only concerned with "core" (and therefore its two dependencies as
well, `network/{shuffle,common}`).

The `mvn package` workflow (and its sbt equivalent) that most people
apparently use involves (for me) compiling+packaging 16 other modules that
I don't care about; I pay this cost whenever I rebase off of master or
encounter the sbt-compiler-crash bug, among other possible scenarios.

Compiling one module (after building/installing its dependencies) seems
like the sort of thing that should be possible, and I don't see why my
previously-documented attempt is failing.

re: Marcelo's comment about "missing the 'spark-parent' project", I saw
that error message too and tried to ascertain what it could mean. Why would
`network/shuffle` need something from the parent project? AFAICT
`network/common` has the same references to the parent project as
`network/shuffle` (namely just a <parent> block in its POM), and yet I can
`mvn install -pl` the former but not the latter. Why would this be? One
difference is that `network/shuffle` has a dependency on another module,
while `network/common` does not.

Does Maven not let you build modules that depend on *any* other modules
without building *all* modules, or is there a way to do this that we've not
found yet?

Patrick: per my response to Marcelo above, I am trying to avoid having to
compile and package a bunch of stuff I am not using, which both `mvn
package` and `mvn install` on the parent project do.

On Tue Dec 02 2014 at 3:45:48 PM Marcelo Vanzin <> wrote:

> On Tue, Dec 2, 2014 at 2:40 PM, Ryan Williams
> <> wrote:
> > Following on Mark's Maven examples, here is another related issue I'm
> > having:
> >
> > I'd like to compile just the `core` module after a `mvn clean`, without
> > building an assembly JAR first. Is this possible?
> Out of curiosity, may I ask why? What's the problem with running "mvn
> install -DskipTests" first (or "package" instead of "install",
> although I generally do the latter)?
> You can probably do what you want if you manually build / install all
> the needed dependencies first; you found two, but it seems you're also
> missing the "spark-parent" project (which is the top-level pom). That
> sounds like a lot of trouble though, for not any gains that I can
> see... after the first build you should be able to do what you want
> easily.
> --
> Marcelo

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message