crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gabriel Reid <>
Subject Re: Moving PType and friends
Date Sun, 28 Oct 2012 12:38:08 GMT

On 28 Oct 2012, at 10:39, Matthias Friedrich <> wrote:

> Hi,
> On Saturday, 2012-10-27, Josh Wills wrote:
>> On Sat, Oct 27, 2012 at 2:41 PM, Matthias Friedrich <> wrote:
> OK, I see the problem. As usual we want the same thing, mostly ;-)
>> If it's possible to spell out exactly what the APIs look like when
>> this is done, let's do that; if we're not there yet but could spell
>> out a set of principles that we would like to be true of the APIs,
>> then let's do that and then start creating some strawman proposals for
>> organization of code that satisfies them as closely as we can.
>> Implementing the changes required to get to the goals may take longer
>> than a single release cycle, but I wouldn't want it to take more than
>> two releases, just because I'm loathe to be in a process of
>> continually moving things around.
> Good idea, let's first agree on a set of principles. In my opinion,
> we should limit the scope for these prinicples to client-facing
> packages, everything else can be changed in any way at any time.
> My proposal is based on [2], a very short and incomplete summary can
> be found at [3]. For us, it boils down to this:
> * A package must have a clear purpose; it contains either mostly
>   abstractions or mostly implementations (this makes it easier
>   to explain)
> * A package must not depend on a package that is less stable 
>   than itself (meaning a package containing mostly abstractions
>   must not depend on one containing mostly implementations)
> * There must be no dependencies from a client-facing package to
>   an internal package (that is, javadocs don't have dangling
>   references)
> * There must be tight cohesion between classes in a package or
>   the package should be split (this doesn't apply for .util)
> * There must be no dependency cycles between client-facing packages
> You can calculate metrics for all of this but it's really just common
> sense. Crunch follows these rules in the vast majority of cases
> already. Right now I see the following violations:
> * The .types package mixes abstractions and implementations and
>   is part of a dependency cycle with base.
> * The base package references the .io implementation package
>   causing a dependency cycle.
> * The base package references the .util package causing a
>   dependency cycle.
> * There are lots of implementations in CombineFn and other Fns
>   that shouldn't be in base (which is for abstractions). We should
>   move them to .fn, perhaps to Guava style CombineFns, FilterFns.
>   We can even do this in a backwards compatible way.

I think that this is exactly what we needed: a clear view on where
we want to go and what it'll get us (thanks for the outline Matthias). 
Following these principles to remove the violations you listed sounds 
good to me.

> Note: Java 8 will be a game changer; as far as I understand it, our
> abstract Fns are *not* going to work with Lambdas, we would need
> interfaces with only a single method. Does anyone think we have to
> address this before Java 8 is released?

I don't think we absolutely *need* to address this, but you can be sure that
I definitely do *want* to address it before the release of Java 8. Seeing the
way that pipelines can be implemented in Scrunch definitely makes me 
jealous. It looks like we've got about 8 or 9 months to have this sorted out.

- Gabriel

View raw message