Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.co.in;
  h=DKIM-Signature:Received:X-Yahoo-SMTP:X-YMail-OSG:X-Yahoo-Newman-Property:Message-ID:From:To:References:Subject:Date:MIME-Version:Content-Type:Content-Transfer-Encoding:X-Priority:X-MSMail-Priority:X-Mailer:X-MimeOLE;
  b=qivFEj02MG6CfiGt4t50zQT1BMZUSQaZ4cXOH9XgwGDOvgTTfZY29DrS6Os1cBlTvNjU2jweJPNvgFcEFdrF/Ihv8UtPoXmxxXKF9I+7doKQmGDi0x2hLl4FvXUzqj0rvNAqqKHLqpX/IsfUufqYsAFrRxff07Fx4DudlX6k/aI=
  ;
Message-ID: <565726F0A1EF4793BDFB7BAC5A44CB81@sv.us.sonicwall.com>
From: "Ganesh" <emailgane@yahoo.co.in>
To: <java-user@lucene.apache.org>
References: <AANLkTi=zbHP+pPjdCirMWFEvQ5jRt7-siH+ZfoyaNE4o@mail.gmail.com>
 <AANLkTi=BqxCW8OEFJcAGqtK3dyhROOR=nifwObAx5a9=@mail.gmail.com>
 <AANLkTik-Mag0ZHC9Y6gLKmU2ODORxfACDO6V514J1wmm@mail.gmail.com>
 <E5331E53F021E443B53EC78F375B3C761F9424@buf-ex02.cymfony.com>
 <AANLkTi=XvkmzuEtHgYVkhF-yyt3i--BMHn2f_gZfpWgx@mail.gmail.com>
 <E5331E53F021E443B53EC78F375B3C761F943B@buf-ex02.cymfony.com>
 <00e401cb8a60$f390d5c0$dab28140$@thetaphi.de>
 <AANLkTinNd27F8_PTBvhmeiErNo68JqNVyGsK5ZpjmaB0@mail.gmail.com>
 <AANLkTi=Zkv3Q5Jmc38gqjuN6woPAUur9w+w93WSkC20T@mail.gmail.com>
 <100E98B677A547C3BE16871066C3614D@sv.us.sonicwall.com>
 <001301cb8c76$8af2d230$a0d87690$@thetaphi.de>
 <AANLkTiktmOUv-pEHhRfcSpUsKT_f=8LF=Apd2ujEmeh1@mail.gmail.com>
Subject: Re: best practice: 1.4 billions documents
Date: Thu, 25 Nov 2010 14:25:05 +0530
MIME-Version: 1.0
Content-Type: text/plain;
	charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

Thanks for the input.

My results are sorted by date and i am not much bothered about score. =
Will i still be in trouble?

Regards
Ganesh


----- Original Message -----=20
From: "Robert Muir" <rcmuir@gmail.com>
To: <java-user@lucene.apache.org>
Sent: Thursday, November 25, 2010 1:45 PM
Subject: Re: best practice: 1.4 billions documents


On Thu, Nov 25, 2010 at 2:58 AM, Uwe Schindler <uwe@thetaphi.de> wrote:
> ParallelMultiSearcher as subclass of MultiSearcher has the same =
problems. These are not crashes, but more that some queries do not =
return correct scored results for some queries. This effects especially =
all MultiTermQueries (TermRange, Fuzzy, NumericRange, Wildcard, Prefix) =
if they are used in a negative fashion (using MUST_NOT resp. "-" in =
QueryParser). For all of those queries except Fuzzy, you are safe if you =
use CONSTANT_SCORE_REWRITE_METHOD (using setRewriteMethod). The same =
problems apply for span queries. For *all* Fuzzy Queries (negative or =
not), the scores are simply wrong and so scoring is broken with =
(Parallel)MultiSearcher; wrong results are only returned when negative =
clauses!
>

you can use constant score rewrite method with fuzzy, too. then it
will work "correctly" (even negative) with multisearcher too. but it
will be slow, with unbounded number of results, and the fuzziness will
not affect the scoring. (this is what constant score rewrite implies)

the reason i say "correctly" is that for all of these queries,
constant score rewrite is just a general workaround, and might still
be incorrect. This is because many queries often have special cases
where they rewrite to simpler things and in general the MultiSearcher
combine() logic is broken here, so there might be more problems.

> A new class ParallelIndexSearcher could help with that, when it =
parallelizes multiple segments, this is still in planning phase. The =
difference to ParallelMultiSearcher would be that it takes a "single" =
IndexReader (e.g. a MultiReader in your case) and parallelizes per =
segment/segment bunches.
>

Besides the inherited broken-ness from multisearcher,
parallelmultisearcher is broken further because it requires you to
organize your index structure in a special way to get concurrency.

This is all pretty silly though, since ParallelMultiSearcher on a
single machine isn't going to increase QPS, so how useful really is it
in general???

we should deprecate both the broken Multi & ParallelMulti Searchers
and never look back.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Send free SMS to your Friends on Mobile from your Yahoo! Messenger. Download Now! http://messenger.yahoo.com/download.php

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org