Return-Path: Delivered-To: apmail-lucene-openrelevance-user-archive@minotaur.apache.org Received: (qmail 3840 invoked from network); 11 Feb 2010 21:32:10 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 11 Feb 2010 21:32:10 -0000 Received: (qmail 39369 invoked by uid 500); 11 Feb 2010 21:32:10 -0000 Delivered-To: apmail-lucene-openrelevance-user-archive@lucene.apache.org Received: (qmail 39321 invoked by uid 500); 11 Feb 2010 21:32:10 -0000 Mailing-List: contact openrelevance-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: openrelevance-user@lucene.apache.org Delivered-To: mailing list openrelevance-user@lucene.apache.org Received: (qmail 39312 invoked by uid 99); 11 Feb 2010 21:32:10 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2010 21:32:10 +0000 X-ASF-Spam-Status: No, hits=2.2 required=10.0 tests=HTML_MESSAGE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of mbennett.ideaeng@gmail.com designates 209.85.223.196 as permitted sender) Received: from [209.85.223.196] (HELO mail-iw0-f196.google.com) (209.85.223.196) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 11 Feb 2010 21:32:01 +0000 Received: by iwn34 with SMTP id 34so1786009iwn.21 for ; Thu, 11 Feb 2010 13:31:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:from:date:x-google-sender-auth:message-id:subject:to :content-type; bh=gM5fGXC6MQkQrfyDuulZGfniGRZSHZCVxt4C9DWyjI4=; b=R140jlZ6TQ7T57dk5EBkX6JTU+Nvq8LAoS85XxS+hIw/6bJ3eXvkh79ZfzZNmzzUyt d0ptIwxqJuY12wVh+YGNQm4gUAGofSL4k5N/kZOWpwYt7soBLezXQbskaq1nuVWgPepw JLt2op/LysOowCrhU5VGk22DuKEjp8mZntUNo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:content-type; b=kHT4XIGc9c3IMIiblhWDWdsteUNvGSiGj5pqeiUJQ8piwolpF49S3GIDrQOZa8L97j FZxsSTzVlRPdyDibD36hNkmM6OwOBZ8SiR80ZBq/uDxFzz9hwmi7d3SA07GOTaPba68K IiMZLvanzjucM0GGBqvPoGtGNBSbfvNNcHFPg= MIME-Version: 1.0 Sender: mbennett.ideaeng@gmail.com Received: by 10.231.154.213 with SMTP id p21mr713373ibw.42.1265923900315; Thu, 11 Feb 2010 13:31:40 -0800 (PST) In-Reply-To: <8f0ad1f31002111302m29d17ac9v4138b29abbd64feb@mail.gmail.com> References: <3504767f1002110958y1d12650h8c6b22c062f5f271@mail.gmail.com> <8f0ad1f31002111149t2795b6d8p848668f3d1171c77@mail.gmail.com> <3504767f1002111249s1b11046dweefd3c4bb55032f1@mail.gmail.com> <8f0ad1f31002111302m29d17ac9v4138b29abbd64feb@mail.gmail.com> From: Mark Bennett Date: Thu, 11 Feb 2010 13:31:20 -0800 X-Google-Sender-Auth: aa9805da7485966c Message-ID: <3504767f1002111331j26a4afa5h41596da65db9e294@mail.gmail.com> Subject: Re: Comments on ORP Wiki Additions ? To: openrelevance-user@lucene.apache.org Content-Type: multipart/alternative; boundary=0016e68ee03964171a047f59e0b2 X-Virus-Checked: Checked by ClamAV on apache.org --0016e68ee03964171a047f59e0b2 Content-Type: text/plain; charset=ISO-8859-1 Hi Robert, By "pooling", you mean they combine different sets of source docs and question sets, in kind of a patch work? If that's what you mean, do you know how that process was generally done? How close to "perfection", ie total coverage by humans, do you think they got? If that's not what you meant by "pooling" then I'm a bit confused... Thanks, Mark -- Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513 On Thu, Feb 11, 2010 at 1:02 PM, Robert Muir wrote: > in this case pooling is what is typically used. > > > On Thu, Feb 11, 2010 at 3:49 PM, Mark Bennett wrote: > >> Thanks Robert, >> >> Excellent comments, I'll try to add something to the outline. Either a >> higher level top section, or some intro text. >> >> Robert, in particular, I wonder if you could look at: >> >> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing >> >> In the section on "Full-Grid Assertions (TREC-Style!)" >> >> It talks about the "M x N" problem of creating relevancy judgment data. >> It also explores some of the shortcuts that could be used. >> >> We're actually working through these problems with a couple clients. On >> the one hand they want "perfect" measurements, but on the other hand nobody >> wants to fund the work to create completely curated test sets. This is the >> classic "good vs. cheap" argument, and I DO think there are reasonable >> compromises to be had. >> >> TREC has evolved over the years and I wonder how they've addressed these. >> Did they take any shortcuts? Or did they get enough manpower to really >> curate every single document and relevancy judgment? >> >> I'll be adding more about some of the compromises we've considered and >> worked on, but it'd be great to get other experts to chime in. Either y'all >> will come back with other ideas we didn't think, or we get to say "we told >> you so" - I'm happy either way. >> >> And what I love about the ORP process is that all of this is captured and >> vetted in an accessible public forum. TREC was also peer reviewed, so this >> continues that tradition in the newer medium. And I'll work on an even >> clearer outline >> >> >> Mark >> >> -- >> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com >> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513 >> >> >> On Thu, Feb 11, 2010 at 11:49 AM, Robert Muir wrote: >> >>> first of all, thanks for adding this content! >>> >>> in my opinion one thing that might be helpful would be an 'introduction' >>> section that is VERY high-level. I don't want to sound negative but your >>> 'high level outline' is actually quite technical :) >>> >>> it might be a good thing for this project if we had some content >>> somewhere that explained at a very very high level what this whole relevance >>> testing thing is all about... >>> >>> >>> On Thu, Feb 11, 2010 at 12:58 PM, Mark Bennett wrote: >>> >>>> Good morning Relevancy comrades, >>>> >>>> I've tried to take a stab at outlining this rather complex subject in >>>> the wiki. Of course it's a work in progress. >>>> >>>> I've done a high level outline here: >>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline >>>> >>>> And an expansion of the first section of the outline here: >>>> >>>> http://cwiki.apache.org/confluence/display/ORP/Relevancy+Assertion+Testing >>>> >>>> I actually could use some feedback. I promise you this is not vanity, >>>> there are actually some very pragmatic motives for my postings. >>>> >>>> I guess some specific questions: >>>> * I'm trying to create a bit of a "crash course" in Relevancy Testing, >>>> are there major areas I've overlooked? >>>> * I've outlined 2 broad categories of testing, do you agree? >>>> * I've tried to explore some of the high level strengths and drawbacks >>>> of certain methodologies >>>> * Is the "tone" reasonably neutral? What I mean is that some folks may >>>> be attached to certain methods, I don't want to seem like I'm "trashing" >>>> anything, just trying to point out the strengths and weaknesses in a fair >>>> way. >>>> >>>> I look forward to any comments. >>>> >>>> Mark >>>> >>>> -- >>>> Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com >>>> Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513 >>>> >>> >>> >>> >>> -- >>> Robert Muir >>> rcmuir@gmail.com >>> >> >> > > > -- > Robert Muir > rcmuir@gmail.com > --0016e68ee03964171a047f59e0b2 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hi Robert,

By "pooling", you mean they combine different s= ets of source docs and question sets, in kind of a patch work?=A0 If that&#= 39;s what you mean, do you know how that process was generally done?=A0 How= close to "perfection", ie total coverage by humans, do you think= they got?

If that's not what you meant by "pooling" then I'm a = bit confused...

Thanks,
Mark

--
Mark Benn= ett / New Idea Engineering, Inc. / = mbennett@ideaeng.com
Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513


On Thu, Feb 11, 2010 at 1:02 PM, Robert = Muir <rcmuir@gmail= .com> wrote:
in this case pooling is what is typically used.


On Thu, Feb 11, 2010 at 3:49 PM,= Mark Bennett <mbennett@ideaeng.com> wrote:
Thanks Robert,
Excellent comments, I'll try to add something to the outline.=A0 E= ither a higher level top section, or some intro text.

Robert, in particular, I wonder if you could look at:In the section on "Full-Grid Assertions (TREC-Style!)"<= br clear=3D"all">
It talks about the "M x N" problem of creati= ng relevancy judgment data.=A0 It also explores some of the shortcuts that = could be used.

We're actually working through these problems with a couple clients= .=A0 On the one hand they want "perfect" measurements, but on the= other hand nobody wants to fund the work to create completely curated test= sets.=A0 This is the classic "good vs. cheap" argument, and I DO= think there are reasonable compromises to be had.

TREC has evolved over the years and I wonder how they've addressed = these.=A0 Did they take any shortcuts?=A0 Or did they get enough manpower t= o really curate every single document and relevancy judgment?

I'll be adding more about some of the compromises we've considered = and worked on, but it'd be great to get other experts to chime in.=A0 E= ither y'all will come back with other ideas we didn't think, or we = get to say "we told you so" - I'm happy either way.

And what I love about the ORP process is that all of this is captured a= nd vetted in an accessible public forum.=A0 TREC was also peer reviewed, so= this continues that tradition in the newer medium.=A0 And I'll work on= an even clearer outline


Mark

--
Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com=
Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513


On Thu, Feb 1= 1, 2010 at 11:49 AM, Robert Muir <rcmuir@gmail.com> wrote:
first of all, thanks for adding this content!

in my opinion one thin= g that might be helpful would be an 'introduction' section that is = VERY high-level. I don't want to sound negative but your 'high leve= l outline' is actually quite technical :)

it might be a good thing for this project if we had some content somewh= ere that explained at a very very high level what this whole relevance test= ing thing is all about...


On Thu, Feb 11, 2010 at 12:58 PM, Mark Bennett <mbennett@ideaeng.com> wrote:
Good morning Rele= vancy comrades,

I've tried to take a stab at outlining this rath= er complex subject in the wiki.=A0 Of course it's a work in progress.
I've done a high level outline here:
http://cwiki.apache.org/confluence/display/ORP/Relevancy+Testing+Outline

And an expansion of the first section of the outline here:
http://cwiki.apache.org/confluence/display/ORP/Relev= ancy+Assertion+Testing

I actually could use some feedback.=A0 I promise you this is not vanity= , there are actually some very pragmatic motives for my postings.

I = guess some specific questions:
* I'm trying to create a bit of a &qu= ot;crash course" in Relevancy Testing, are there major areas I've = overlooked?
* I've outlined 2 broad categories of testing, do you agree?
* I'= ;ve tried to explore some of the high level strengths and drawbacks of cert= ain methodologies
* Is the "tone" reasonably neu= tral?=A0 What I mean is that some folks may be attached to certain methods,= I don't want to seem like I'm "trashing" anything, just = trying to point out the strengths and weaknesses in a fair way.

I look forward to any comments.

Mark
=
--
Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
Direct: 40= 8-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513



--
Robert Muir
rcmuir@gmail.com




--
Robert Muir
rcmuir@gmail.com

--0016e68ee03964171a047f59e0b2--