Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of erickerickson@gmail.com
 designates 64.233.182.187 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=k+2SivYePsUc1cw6OfXFKc6YOhkJft4aLc07MdpaOL4CKf/rTfv/rQdwx440LChiarr/418U6OreGAKrMkV9k5ed8STb7xNljkBtxrBneisjW8K5V7Il9Jt7LtQ9kFVZaPXH93NoWXoSA3dETtMZtC/5M3mDKzAnc5mLEPsrBCY=
Message-ID: <359a92830711200756t7005280bx4d2888f886a30c1e@mail.gmail.com>
Date: Tue, 20 Nov 2007 10:56:28 -0500
From: "Erick Erickson" <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Is storing 20 fields in a lucene document desirable?
In-Reply-To: <C8DCFA59-9540-4DE1-8677-62FE41CD7192@apache.org>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_18886_23706984.1195574188804"
References: <13855272.post@talk.nabble.com>
	 <C8DCFA59-9540-4DE1-8677-62FE41CD7192@apache.org>

------=_Part_18886_23706984.1195574188804
Content-Type: text/plain; charset=WINDOWS-1252
Content-Transfer-Encoding: quoted-printable
Content-Disposition: inline

How are you doing your search? When you say "lucene is the
bottleneck", that encompasses a lot. You really need to pinpoint things
a bit more....

1> are you iterating over the hits object for many docs? This is bad.

2> are you using a HitCollector and reading the doc each time you get
to the collect method? This is bad.

3> Is it the *search* or the fetch that's eating the time? Time *just* the
search call (making sure it doesn't go through a HitCollector) to answer
this.

4> How many documents are you returning? 5? 5,000? the answer may
tell us a lot.

5> are you opening a new reader each time you search? this is bad.

6> Have you looked over
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed?

Before you try and implement an solution, you *must* figure out what
operation is
actually taking the time. an 8G index isn't that big, so I suspect that
there's
something else going on. What kind of throughput are you seeing? What
do you need? Can you post some sample timings? Can you post some sample
search code?

I can't tell you how many times I've been *sure* I knew what the problem
was, but
in the process of analyzing the problem discovered that my real issue was
something totally different.....

Best
Erick

On Nov 20, 2007 7:21 AM, Grant Ingersoll <gsingers@apache.org> wrote:

>
> On Nov 20, 2007, at 6:29 AM, kumarlimbu wrote:
>
> >
> > Our document contains a total of 23 fields in one document and we
> > STORE all
> > of them in lucene index.
> >
> > We have recently had some performance issues and our analysis has
> > shown the
> > bottleneck to be lucene search and retrieval.
>
> Perhaps you can share your information on java-dev along with any
> detailed tests, etc. so that we can see if there is anything we can
> improve.
>
> >
> >
> > We have been thinking about reducing the number of fields per
> > document by
> > removing unnecessary fields and by merging fields with similar
> > weightings.
> > Will reducing the number of fields help to optimize performance?
>
> Yes
>
> >
> >
> > Another issue is we are currently retrieving around 9 fields after
> > we do a
> > search. Some are long text of up to 1000 words. Is it a large
> > overhead to
> > retrieve long fields?
>
> Yes.  Are you using FieldSelector?  Also, I would only STORE those
> fields that you actually need to display, not all 23.  Do you display
> all 9 fields right away or are some only when you choose a document?
> If so, try the Lazy Loading piece of FieldSelector.
>
> >
> >
> > We are considering the option of separating the search and retrieve
> > parts so
> > that Lucene performs the search, MYSQL stores the data. We just
> > store the
> > INDEXED field and primary key in the lucene index. After searching
> > we only
> > return 1 field (primary key) instead of 9 fields. This field will be
> > used
> > for retrieving the actual information from the MySQL database. Will
> > reducing
> > the number of fields retrieved from lucene reduce the response time
> > or will
> > using MySQL database make it worse?
>
> Hard to say, you will likely have to try it out.
>
> >
> >
> > So our main concern is to find if retrieving fields usually takes
> > longer
> > than searching or not? What does lucene spend most time doing =96
> > search or
> > retrieval? We are also concerned that using MySQL will have
> > performance
> > issues because we will be doing I/O for MySQL as well as Lucene. We
> > also add
> > around 100k documents each day and remove around the same number of
> > documents. Will this frequent read and write have impact on
> > performance?
> >
>
> What version of Lucene are you using?  Naturally, if you are updating
> things that will have an effect, but that may not be a factor.  I
> would check out http://wiki.apache.org/lucene-java/BasicsOfPerformance
>
> Also, you may want to try out (not in production) the trunk version of
> Lucene.
>
> Last, but not least, do you have some sort of cache for your
> Documents?  Obviously, you need to have appropriate semantics for when
> the underlying docs change, but using a cache makes sense, too.
>
> -Grant
>
>
> --------------------------
> Grant Ingersoll
> http://lucene.grantingersoll.com
>
> Lucene Helpful Hints:
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

------=_Part_18886_23706984.1195574188804--