Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of prasen.bea@gmail.com
 designates 209.85.212.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=WXSKD8yRX4Vp3Si2YdMn9nalSjVzNA7xjVBon/s5WqVU/QdyYZhXbFR/hqNjG6yIV1
         FVipZdYmAHbCMwFIfkybF3TPjcL39sm3q6D7UNSdZSXi+cmLxElwkGTaT1OkwQgyaWY8
         klnBeDNRO0zTW0G6X63mRflAPmAE0bz7vZi7Y=
MIME-Version: 1.0
In-Reply-To: <o2r279092301004011840r48ad87adm9a2598698cfb681f@mail.gmail.com>
References: <o2r279092301004011840r48ad87adm9a2598698cfb681f@mail.gmail.com>
Date: Fri, 2 Apr 2010 07:35:27 -0400
Message-ID: <t2y500e4a1e1004020435l5f7d921dx196f73c10fd46959@mail.gmail.com>
Subject: Re: Lucene Challenge - sum, count, avg, etc.
From: prasenjit mukherjee <prasen.bea@gmail.com>
To: java-user@lucene.apache.org, jeacott@hardlight.com.au
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Pig generally takes csv-type flat files as input. And then you do
join/group-by/sum/count etc on the variables ( aka relations )

For Michael's example with following data:

*Affiliate / SaleDate / SaleAmount*
* mike / 2010-03-01 / 10.00
* john / 2010-03-01 / 10.00

One can write following pig-script :

r1 =3D load 'data.csv' as ( affiliate:string, saledate:string, amount:int )
r1 =3D filter r1 by saledate > my_udf_convert_in_secs(2010-03-01) AND
saledate < my_udf_convert_in_secs(2010-03-06)
r2 =3D group r1 by affiliate;
r3 =3D foreach r2 generate affiliate, SUM(amount) as totalrevenue;
r3 =3D order r3 by totalrevenue;
DUMP r3 into 'output.csv'

Remember that its only because of the additional sorting requirement
that we are forced to use pig, otherwise lucene can do the job (
except the sorting ) much faster.

-Prasen

On Thu, Apr 1, 2010 at 9:40 PM, Jason Eacott <jason@hardlight.com.au> wrote=
:
> Thanks for the ref - didn't know about Pig before.
> the language and approach looks useful, so now I'm wondering if it
> couldn't be used
> across lucene over hadoop too. If data was indexed in lucene and Pig knew=
 that,
> then it could make for an interesting alternate lucene query language.
>
> could this work?
>
>
> prasenjit mukherjee wrote:
>> This looks like a use case more suited =A0for Pig ( over Hadoop ).
>>
>> It could be difficult for lucene to do sort and sum simultaneously as
>> sorting itself depends upon summed value.
>>
>> On Thu, Apr 1, 2010 at 11:47 PM, Michel Nadeau <akaris@gmail.com> wrote:
>>> Well that's my problem: we have a lot of records of all types (afiiliat=
es,
>>> sales) so looping tons of records each time isn't possible.
>>>
>>> - Mike
>>> akaris@gmail.com
>>>
>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org