lucene-openrelevance-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Omar Alonso <oralo...@yahoo.com>
Subject Re: OpenRelevance and crowdsourcing
Date Sat, 17 Oct 2009 00:30:18 GMT
Sure.

1- We can start by paying between 2 and 5 cents per document/query pair (or document/topic)
on a short data set (say 200 docs). That should be in the order of $25 (assuming 2 cents and
5 turkers per assignment + AMZN fee). 

It also depends how many experiments one would like to run. My suggestion would be to run
2 or 3 experiments with some small data sets for say $100 to see what kind of response we
get back and then think about something else at large scale. 

I have some tips on how to run crowdsourcing for relevance evaluation here: http://wwwcsif.cs.ucdavis.edu/~alonsoom/ExperimentDesign.pdf

2- If the goal is to have everything open source (gold set + relevance judgments), we need
to produce a new data set from scratch. Also, what is the goal here? What is the domain? Enterprise
search? Ad-hoc retrieval? 

In summary, I would start with something small (English only, Creative Commons or Wikipedia).
Build a few experiments and see the results. Then expand on data sets and also make it multilingual.

o.

--- On Fri, 10/16/09, Grant Ingersoll <gsingers@apache.org> wrote:

> From: Grant Ingersoll <gsingers@apache.org>
> Subject: Re: OpenRelevance and crowdsourcing
> To: openrelevance-dev@lucene.apache.org
> Cc: openrelevance-user@lucene.apache.org
> Date: Friday, October 16, 2009, 3:38 PM
> Hi Omar,
> 
> It sounds interesting, can you elaborate more on what you
> had in mind?
> 
> A few questions come to mind:
> 
> 1. Cost associated w/ Turk.
> 2. What dataset would you use?
> 
> -Grant
> 
> On Oct 16, 2009, at 5:49 PM, Omar Alonso wrote:
> 
> > Hello,
> > 
> > I would like to know if there is interest in trying
> some experiments on Mechanical Turk for the OpenRelevance
> project. I've done TREC and INEX on MTurk and is a good
> platform for trying relevance experiments.
> > 
> > Regards,
> > 
> > Omar
> > 
> > 
> > 
> 
> 
> 


      

Mime
View raw message