lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From prasenjit mukherjee <prasen....@gmail.com>
Subject Re: Lucene Challenge - sum, count, avg, etc.
Date Fri, 02 Apr 2010 11:35:27 GMT
Pig generally takes csv-type flat files as input. And then you do
join/group-by/sum/count etc on the variables ( aka relations )

For Michael's example with following data:

*Affiliate / SaleDate / SaleAmount*
* mike / 2010-03-01 / 10.00
* john / 2010-03-01 / 10.00

One can write following pig-script :

r1 = load 'data.csv' as ( affiliate:string, saledate:string, amount:int )
r1 = filter r1 by saledate > my_udf_convert_in_secs(2010-03-01) AND
saledate < my_udf_convert_in_secs(2010-03-06)
r2 = group r1 by affiliate;
r3 = foreach r2 generate affiliate, SUM(amount) as totalrevenue;
r3 = order r3 by totalrevenue;
DUMP r3 into 'output.csv'

Remember that its only because of the additional sorting requirement
that we are forced to use pig, otherwise lucene can do the job (
except the sorting ) much faster.

-Prasen

On Thu, Apr 1, 2010 at 9:40 PM, Jason Eacott <jason@hardlight.com.au> wrote:
> Thanks for the ref - didn't know about Pig before.
> the language and approach looks useful, so now I'm wondering if it
> couldn't be used
> across lucene over hadoop too. If data was indexed in lucene and Pig knew that,
> then it could make for an interesting alternate lucene query language.
>
> could this work?
>
>
> prasenjit mukherjee wrote:
>> This looks like a use case more suited  for Pig ( over Hadoop ).
>>
>> It could be difficult for lucene to do sort and sum simultaneously as
>> sorting itself depends upon summed value.
>>
>> On Thu, Apr 1, 2010 at 11:47 PM, Michel Nadeau <akaris@gmail.com> wrote:
>>> Well that's my problem: we have a lot of records of all types (afiiliates,
>>> sales) so looping tons of records each time isn't possible.
>>>
>>> - Mike
>>> akaris@gmail.com
>>>
>>>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message