Mailing-List: contact dev-help@spark.incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@spark.incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of ctn@adatao.com designates
 209.85.223.182 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAHGoR3-K3HNsNx1jHVm7nHanicRL7hiWkz3QJVWgB0o+KNXvyw@mail.gmail.com>
References: 
 <CAHGoR3-K3HNsNx1jHVm7nHanicRL7hiWkz3QJVWgB0o+KNXvyw@mail.gmail.com>
From: Christopher Nguyen <ctn@adatao.com>
Date: Sat, 12 Oct 2013 14:22:16 -0700
Message-ID: 
 <CAGh_TuP0iS40Xav3bQBmRJx9U4-q7dsXTdwOYZzpefJwpy=a-g@mail.gmail.com>
Subject: Re: Test coverage of Spark
To: dev@spark.incubator.apache.org
Content-Type: multipart/alternative; boundary=089e0111e074867cd404e891d59d

--089e0111e074867cd404e891d59d
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Roman, an area I think would (a) have high impact, and (b) is relatively
not well covered is performance analysis. I'm sure most teams are doing
this internally at their respective companies, but there is no shared code
base and shared wisdom about what we're finding/improving.

For example, consider the task of loading a table from disk into memory by
Shark. We're getting conflicting data about how much of this is cpu-bound
vs I/O-bound. Our effort to track this down should be sharable somehow, and
would benefit from others' findings. Of course this is dependent on the
particular configuration, but there is a lot of test harness code/scripts
that can be shared. And individual findings, even if/especially if they are
conflicting, are very valuable if well documented.

There is a Benchmark effort covered here
https://amplab.cs.berkeley.edu/benchmark/, but it addresses a slightly
different goal. You could consider this Perf-Analysis as part of that, or
as its own effort.

This may be more than you were looking to own, but given your stated
enthusiasm :) I want to throw the idea out there.

--
Christopher T. Nguyen
Co-founder & CEO, Adatao <http://adatao.com>
linkedin.com/in/ctnguyen


On Sat, Oct 12, 2013 at 1:48 PM, =D0=A0=D0=BE=D0=BC=D0=B0=D0=BD =D0=A2=D0=
=BA=D0=B0=D0=BB=D0=B5=D0=BD=D0=BA=D0=BE <tkalenkoroman@gmail.com>wrote:

> Hello.
> I'm trying to dive into Spark's sources on a deeper-than-mere-glance leve=
l
> and I find beginning with writing unit tests a good way to do it. So,
> basically, I'm wondering if there are points to which I could specificall=
y
> apply my enthusiasm, i. e. are there some un- or not enough covered parts
> for which I could write some tests?
> I'm wondering as well about the state of Apache-hosted JIRA for Spark - I
> currently can't see any entry in there. Should I look for them in Github
> mirror or still in the antecedent JIRA instance on
> http://spark-project.atlassian.net/?
> Regards,
> Roman.
>

--089e0111e074867cd404e891d59d--