I completely agee with this.  In practice, search engines and to a larger
extent recommendation engines shape user behavior and are, in turn, shaped
by user behavior so that static relevancy tests are of only very limited
value in the end game.

But it is still *very* nice to have them.

On Mon, May 18, 2009 at 8:00 PM, Mark Miller <markrmiller@gmail.com> wrote:

> Grant Ingersoll wrote:
>
>> Some interesting discussion at
>> http://thenoisychannel.com/2009/05/18/copying-trec-is-the-wrong-track-for-the-enterprise/
>>
> That was an interesting read. I think a lot of the argument misses the
> point. It doesn't seem to me that the main benefit or intent comes from
> 'bake offs' with other search engines ("Selling search applications to
> enterprises isn't, in my experience, about winning relevance bake-offs.") -
> the main benefit is allowing us to measure changes and improvements to
> Lucene's relevancy calculations and to make judgments about how Lucene
> currently performs. I see it easily as important as the Lucene benchmark
> contrib. Its not going to be a secret sauce, just like the benchmarker has
> been no secret sauce - but its going to make it easier to reliably improve
> Lucene in the future.
>
>