Mailing-List: contact dev-help@spark.apache.org; run by ezmlm
Precedence: bulk
Received-SPF: pass (nike.apache.org: domain of vanzin@cloudera.com designates
 209.85.216.172 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CANeJXFORRC0rOSq5KJuWURCVNeMPE6JT5krE9E+kHP4ZQh5QgA@mail.gmail.com>
References: 
 <CANeJXFPNU5kweJ2ftsG_bZbqqJ-v=TG5FkdvuENiFdWUJAYabQ@mail.gmail.com>
	<C09AD697-30E7-471E-8F14-AF5814EA216D@gmail.com>
	<CAAsvFP=V9FL=KvXNUeWVfFz8q04oj1exdwSaSNiab5Vc9hFkUg@mail.gmail.com>
	<CANeJXFMRj2JdyrAQsBvyK3s3t+R1_QBeDSzqQ-OKUC2WhdfSvQ@mail.gmail.com>
	<CAAOnQ7tAEWS5dEX4bG6ik6j+820er3RCokwP3+A9caFyExDrdg@mail.gmail.com>
	<CANeJXFORRC0rOSq5KJuWURCVNeMPE6JT5krE9E+kHP4ZQh5QgA@mail.gmail.com>
Date: Tue, 2 Dec 2014 15:46:19 -0800
Message-ID: 
 <CAAOnQ7vggDLGyH4oVpZw9UzgcAs8JmwVymOe2dTc0HxijQ0rrw@mail.gmail.com>
Subject: Re: Spurious test failures, testing best practices
From: Marcelo Vanzin <vanzin@cloudera.com>
To: Ryan Williams <ryan.blake.williams@gmail.com>
Cc: "dev@spark.apache.org" <dev@spark.apache.org>
Content-Type: text/plain; charset=UTF-8

On Tue, Dec 2, 2014 at 3:39 PM, Ryan Williams
<ryan.blake.williams@gmail.com> wrote:
> Marcelo: by my count, there are 19 maven modules in the codebase. I am
> typically only concerned with "core" (and therefore its two dependencies as
> well, `network/{shuffle,common}`).

But you only need to compile the others once. Once you've established
the baseline, you can just compile / test "core" to your heart's
desire. Core tests won't even run until you build the assembly anyway,
since some of them require the assembly to be present.

Also, even if you work in core - I'd say especially if you work in
core - you should still, at some point, compile and test everything
else that depends on it.

So, do this ONCE:

  mvn install -DskipTests

Then do this as many times as you want:

  mvn -pl spark-core_2.10 something

That doesn't seem too bad to me. (Be aware of the "assembly" comment
above, since testing spark-core means you may have to rebuild the
assembly from time to time, if your changes affect those tests.)

> re: Marcelo's comment about "missing the 'spark-parent' project", I saw that
> error message too and tried to ascertain what it could mean. Why would
> `network/shuffle` need something from the parent project?

The "spark-parent" project is the main pom that defines dependencies
and their version, along with lots of build plugins and
configurations. It's needed by all modules to compile correctly.

-- 
Marcelo

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org