Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of teddyyyy123@gmail.com
 designates 209.85.214.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAAHmpkgiuoR-G6f7WGtGZ55yd2x8Q13zorMW6dMnAvW7KcxBiw@mail.gmail.com>
References: 
 <CAAnh3_8iE902V7Nbj_BJcPpi9DMRvHVjbN_Z1+emxUS9YLth8g@mail.gmail.com>
 <CAFAd71U+tV7CP43yrjFQ9SzeHSXTohJRTZWfRjag94M-NMVMQw@mail.gmail.com>
 <CAFAd71Wa9JiGt1VUy1mDD9EP3Jk9VR2S458P5XbOnUJEG6F7HA@mail.gmail.com>
 <CAEY5pxWCXFoFqdCT2Z4KGhRr6ZQGjamSk10ouzY5bYVVqvbKCA@mail.gmail.com>
 <CAAnh3_-7NZf9h33fNi=RSyFrSM+YaFnS-En0bhTXJHuW0gNY5w@mail.gmail.com>
 <CAAnh3_-uFCR+PSOnBgHhK74yR3Ea3S4AXeyQiKOu7tc-dLAY2A@mail.gmail.com>
 <CAAHmpkgiuoR-G6f7WGtGZ55yd2x8Q13zorMW6dMnAvW7KcxBiw@mail.gmail.com>
From: Yang <teddyyyy123@gmail.com>
Date: Sat, 26 May 2012 07:32:45 -0700
Message-ID: 
 <CAAnh3__0AbJieRYt82Pd7Saf4=ivbGj9OEsN1-ctmDOsk8scSw@mail.gmail.com>
Subject: Re: lucene (search) performance tuning
To: java-user@lucene.apache.org, simon.willnauer@gmail.com
Content-Type: multipart/alternative; boundary=e89a8ff1c2e8f451d904c0f15ca2

--e89a8ff1c2e8f451d904c0f15ca2
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

I'm using disjunction (OR) query. unfortunately all of the clauses are
optional

On Sat, May 26, 2012 at 4:38 AM, Simon Willnauer <
simon.willnauer@googlemail.com> wrote:

> On Sat, May 26, 2012 at 2:59 AM, Yang <teddyyyy123@gmail.com> wrote:
> > I tested with more threads / processes. indeed this is completely
> > cpu-bound, since running 1 thread gives the same latency as 4 threads (=
my
> > box has 4 cores)
> >
> >
> > given this, is there any way to simplify the scoring computation (i'm
> only
> > using lucene as a first level "rough" search, so the search quality is
> not
> > a huge issue here) , so that, for example, fewer fields are evaluated o=
r
> a
> > simpler scoring function is used?
>
> are you using disjunction or conjunction queries? Can you make some
> parts of the query mandatory?
>
> simon
> >
> > thanks
> > Yang
> >
> > On Fri, May 25, 2012 at 5:47 PM, Yang <teddyyyy123@gmail.com> wrote:
> >
> >> thanks a lot guys
> >>
> >>
> >> On Tue, May 22, 2012 at 1:34 AM, Ian Lea <ian.lea@gmail.com> wrote:
> >>
> >>> Lots of good tips in
> >>> http://wiki.apache.org/lucene-java/ImproveSearchingSpeed, linked from
> >>> the FAQ.
> >>>
> >>>
> >>> --
> >>> Ian.
> >>>
> >>>
> >>> On Tue, May 22, 2012 at 2:08 AM, Li Li <fancyerii@gmail.com> wrote:
> >>> > something wrong when writing in my android client.
> >>> > if RAMDirectory do not help, i think the bottleneck is cpu. you may
> try
> >>> to
> >>> > tune jvm but i do not expect much improvement.
> >>> > the best one is splitting your index into 2 or more smaller ones.
> >>> > you can then use solr s distributed searching.
> >>> > if the cpu is not fully used, yuo can do this in one physical machi=
ne
> >>> >
> >>> > =E5=9C=A8 2012-5-22 =E4=B8=8A=E5=8D=888:50=EF=BC=8C"Li Li" <fancyer=
ii@gmail.com>=E5=86=99=E9=81=93=EF=BC=9A
> >>> >>
> >>> >>
> >>> >> =E5=9C=A8 2012-5-22 =E5=87=8C=E6=99=A84:59=EF=BC=8C"Yang" <teddyyy=
y123@gmail.com>=E5=86=99=E9=81=93=EF=BC=9A
> >>> >>
> >>> >> >
> >>> >> > I'm trying to make my search faster. right now a query like
> >>> >> >
> >>> >> > name:Joe Moe Pizza   address:77 main street  city:San Francisco
> >>> >> >is this a conjunction query or a disjunction query=EF=BC=9F
> >>> >>
> >>> >> > in a index with 20mil such short business descriptions (total si=
ze
> >>> > about 3GB) takes about 100--200ms.
> >>> >> >20m is not a small size, how many results for a query in average=
=EF=BC=9F
> >>> >>
> >>> >> > I profiled the query, most time is spent in TermScorer.score(),
> as is
> >>> > shown by the attached yourkit screenshot.
> >>> >> >that=EF=BC=87s true, for a query, matching and scoring is very ti=
me
> consuming
> >>> > and cpu intensive. another one is io for reading postings.
> >>> >>
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> > I tried loading the index onto tmpfs (in-memory block device), a=
nd
> >>> also
> >>> > tried RAMDirectory, neither helps much.
> >>> >> >if that is true. it seems that io is not the
> >>> >> > I am reading
> >>> > http://www.cnlp.org/presentations/slides/AdvancedLuceneEU.pdf
> >>> >> > it mentions
> >>> >> > Size
> >>> >> > =E2=80=93 Stopword removal
> >>> >> > =E2=80=93 Stemming
> >>> >> > =E2=80=A2 Lucene has a number of stemmers available
> >>> >> > =E2=80=A2 Light versus Aggressive
> >>> >> > =E2=80=A2 May prevent fine-grained matches in some cases
> >>> >> > =E2=80=93 Not a linear factor (usually) due to index compression
> >>> >> >
> >>> >> > so for "stopword removal", I'm already using the standard
> analyzer,
> >>> so
> >>> > stop word removal is already included, right?
> >>> >> >
> >>> >> > also generally any other tricks to try for reducing the search
> >>> latency?
> >>> >> >
> >>> >> > Thanks!
> >>> >> > Yang
> >>> >> >
> >>> >> >
> >>> >> >
> ---------------------------------------------------------------------
> >>> >> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> >> > For additional commands, e-mail: java-user-help@lucene.apache.or=
g
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>>
> >>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--e89a8ff1c2e8f451d904c0f15ca2--