Mailing-List: contact dev-help@crunch.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@crunch.apache.org
Received-SPF: pass (athena.apache.org: domain of davidwhiting@gmail.com
 designates 74.125.82.175 as permitted sender)
MIME-Version: 1.0
Date: Wed, 30 Jul 2014 09:52:38 -0400
Message-ID: 
 <CAGUrFMBHP9HPHr=+uQ5eo-oF2LTLm8EnMt2Hf+MgKbuTnyV05w@mail.gmail.com>
Subject: Use of Iterable with combine in Scrunch
From: David Whiting <davidwhiting@gmail.com>
To: dev@crunch.apache.org
Content-Type: multipart/alternative; boundary=089e01419ffa1e97a204ff697834

--089e01419ffa1e97a204ff697834
Content-Type: text/plain; charset=UTF-8

The Scrunch version of combine accepts a function Iterable[V] => V . This
causes a lot of unexpected behaviour because the iterable that is wrapped
is actually a SingleUseIterable, and much of Scala's collection function
implementations actually try and access the underlying iterator multiple
times if they know that it's possible. This leads to often having to write
code like this:

...
.groupByKey()
.combine { _.iterator reduce { _ + _ } }

This is a silly example of course, because there's an Aggregator for
summation, but if your reduce function is more complex you have to do this
indirection via iterator in order to get correct behaviour.

Possible fixes:
a) Change combine to accept a function TraversableOnce[V] => V or
Iterator[V] => V, better reflecting the single-use nature of the underlying
Iterable
b) Given that most custom combines will in fact be folds over monoids, we
could promote the notion of reduce or fold up the the PGroupedTable itself,
so you can do .groupByKey().foldValues(_+_)

--089e01419ffa1e97a204ff697834--