Mailing-List: contact crunch-dev-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: crunch-dev@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of gabriel.reid@gmail.com
 designates 209.85.212.177 as permitted sender)
Content-Type: text/plain; charset=windows-1252
Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\))
Subject: Re: Moving PType and friends
From: Gabriel Reid <gabriel.reid@gmail.com>
In-Reply-To: <20121028093951.GA19160@mafr.de>
Date: Sun, 28 Oct 2012 13:38:08 +0100
Content-Transfer-Encoding: quoted-printable
Message-Id: <1927DE56-59B4-4106-A54C-9F94EDB3B8A3@gmail.com>
References: <20121027160647.GA9240@mafr.de>
 <E0EDAF16-8798-4BE1-BA0B-FB6DB3C98883@gmail.com>
 <CANb5z2+zv_0sNA+Y7eE4j2Vjz5nBWP9fQndXQjteg4g4xh0apA@mail.gmail.com>
 <20121027214115.GB17856@mafr.de>
 <CANb5z2JDhw9e9kdz2EaY+Vi_VrSOHaKMOR5CojswWCyHfmr_TQ@mail.gmail.com>
 <20121028093951.GA19160@mafr.de>
To: crunch-dev@incubator.apache.org


On 28 Oct 2012, at 10:39, Matthias Friedrich <matt@mafr.de> wrote:

> Hi,
>=20
> On Saturday, 2012-10-27, Josh Wills wrote:
>> On Sat, Oct 27, 2012 at 2:41 PM, Matthias Friedrich <matt@mafr.de> =
wrote:
>=20
[=85]
>=20
> OK, I see the problem. As usual we want the same thing, mostly ;-)
>=20
>> If it's possible to spell out exactly what the APIs look like when
>> this is done, let's do that; if we're not there yet but could spell
>> out a set of principles that we would like to be true of the APIs,
>> then let's do that and then start creating some strawman proposals =
for
>> organization of code that satisfies them as closely as we can.
>> Implementing the changes required to get to the goals may take longer
>> than a single release cycle, but I wouldn't want it to take more than
>> two releases, just because I'm loathe to be in a process of
>> continually moving things around.
>=20
> Good idea, let's first agree on a set of principles. In my opinion,
> we should limit the scope for these prinicples to client-facing
> packages, everything else can be changed in any way at any time.
>=20
> My proposal is based on [2], a very short and incomplete summary can
> be found at [3]. For us, it boils down to this:
>=20
> * A package must have a clear purpose; it contains either mostly
>   abstractions or mostly implementations (this makes it easier
>   to explain)
> * A package must not depend on a package that is less stable=20
>   than itself (meaning a package containing mostly abstractions
>   must not depend on one containing mostly implementations)
> * There must be no dependencies from a client-facing package to
>   an internal package (that is, javadocs don't have dangling
>   references)
> * There must be tight cohesion between classes in a package or
>   the package should be split (this doesn't apply for .util)
> * There must be no dependency cycles between client-facing packages
>=20
> You can calculate metrics for all of this but it's really just common
> sense. Crunch follows these rules in the vast majority of cases
> already. Right now I see the following violations:
>=20
> * The .types package mixes abstractions and implementations and
>   is part of a dependency cycle with base.
> * The base package references the .io implementation package
>   causing a dependency cycle.
> * The base package references the .util package causing a
>   dependency cycle.
> * There are lots of implementations in CombineFn and other Fns
>   that shouldn't be in base (which is for abstractions). We should
>   move them to .fn, perhaps to Guava style CombineFns, FilterFns.
>   We can even do this in a backwards compatible way.

I think that this is exactly what we needed: a clear view on where
we want to go and what it'll get us (thanks for the outline Matthias).=20=

Following these principles to remove the violations you listed sounds=20
good to me.

> Note: Java 8 will be a game changer; as far as I understand it, our
> abstract Fns are *not* going to work with Lambdas, we would need
> interfaces with only a single method. Does anyone think we have to
> address this before Java 8 is released?

I don't think we absolutely *need* to address this, but you can be sure =
that
I definitely do *want* to address it before the release of Java 8. =
Seeing the
way that pipelines can be implemented in Scrunch definitely makes me=20
jealous. It looks like we've got about 8 or 9 months to have this sorted =
out.

- Gabriel