Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (herse.apache.org: domain of askar.zaidi@gmail.com
 designates 66.249.92.174 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=beta;
        h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=dgOhZw+tQ5fB6UOwWasZHdwcW+L/kmYR5r/yeW1GXVAlafYDDriqUuPy84yDEkOGd+LFWEIxPkX9OCjyeHo7UpvZXnPcO+oUuUdSP20ZuQm2Q+n4LFdZ7Vj6eh2r+HjjKD1ApLmhkOceCevxMjFnqUBUMBafhXp1cnlMSx38+2w=
Message-ID: <c0c0e1d40707241428r44ee7926q7f9199687650baf3@mail.gmail.com>
Date: Tue, 24 Jul 2007 17:28:36 -0400
From: "Askar Zaidi" <askar.zaidi@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Fine Tuning Lucene implementation
In-Reply-To: <B6CD8741-E8F5-4D00-8579-DDD8944469AA@apache.org>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_38297_20506843.1185312516806"
References: <c0c0e1d40707241243y5ddfd5fai10381873a2d2bd64@mail.gmail.com>
	 <c0c0e1d40707241331i7be68a3dr27924a8e4f186427@mail.gmail.com>
	 <B6CD8741-E8F5-4D00-8579-DDD8944469AA@apache.org>

------=_Part_38297_20506843.1185312516806
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Thanks for the reply.

I am timing the entire search process with a stop watch, a bit ghetto style.
My getXXX methods are:

Document doc = hits.doc(i);
String str = doc.get("item");

So you can see that I am retrieving the entire document in a search query.
Ideally , I'd like to just retrieve the Field object that I want to run the
search on. I know this will give me a boost as one of my Fields is really
huge.

My query is selecting the entire user data-set in the database. I'd like to
do some SQL based search in the query too so that I pick only those items
where the phrase matches.

Index contains about 650MB of data. Index file size is 14478869 bytes.

thanks,
AZ


On 7/24/07, Grant Ingersoll <gsingers@apache.org> wrote:
>
> Where are you getting your numbers from?  That is, where are your
> timers?  Are you timing the rs.next() loop, or the individual calls
> to Lucene?  What do the getXXXXX methods look like?  How big are your
> queries?  How big is your index?
>
> Essentially, we need more info to really help you.  From what I can
> tell, you are generating 3 different Lucene queries for each record
> in the database.  Frankly, I surprised your slowdown is only linear.
>
> On Jul 24, 2007, at 4:31 PM, Askar Zaidi wrote:
>
> > I have 512MB RAM allocated to JVM Heap. If I double my system RAM
> > from 768MB
> > to say 2GB or so, and give JVM 1.5GB Heap space, will I get quicker
> > results
> > ?
> >
> > Can I expect results which take 1 minute to be returned in 30
> > seconds with
> > more RAM ? Should I also get a more powerful CPU ? A real server class
> > machine ?
> >
> > I have also done some of the optimizations that are mentioned on
> > the Lucene
> > website.
> >
> > thanks,
> > AZ
> >
> > On 7/24/07, Askar Zaidi <askar.zaidi@gmail.com> wrote:
> >>
> >> Hey Guys,
> >>
> >> I just finished up using Lucene in my application. I have data in a
> >> database , so while indexing I extract this data from the database
> >> and pump
> >> it into the index. Specifically , I have the following data in the
> >> index:
> >>
> >> <itemID> <tags> <title> <summary> <contents>
> >>
> >> where itemID is just a number (primary key in the DB)
> >> tags : text
> >> titie: text
> >> summary: text
> >> contents: Huge text (text extracted from files: pdfs, docs etc).
> >>
> >> Now while running a search query I realized that the response time
> >> increases in a linear fashion as the number of <itemID> increase
> >> in the DB.
> >>
> >> If I have 50 items, its 8 seconds
> >> 100 items, its 17 seconds.
> >> 300+ items, its 60 seconds and maybe more.
> >>
> >> In a perfect world, I'd like to search on 300+ items within 10-15
> >> seconds.
> >> Can anyone give me tips to fine tune lucene ?
> >>
> >> Heres a code snippet:
> >>
> >> sql query = "SELECT itemID from items where creator = 'askar' ;
> >>
> >> --execute query--
> >>
> >> while(rs.next()){
> >>
> >> score = doTagSearch(askar,text,itemID);
> >> scoreTitle = doTitleSearch(askar,text,itemID);
> >> scoreSummary = doSummarySearch(askar,text,itemID);
> >>
> >> ----
> >>
> >> }
> >>
> >> So this code asks Lucene to search for the "text" in the itemID
> >> passed.
> >> itemID is already indexed. The while loop will run 300 times if
> >> there are
> >> 300 items....that gets slow...what can I do here ??
> >>
> >> thanks for the replies,
> >>
> >> AZ
> >>
>
> --------------------------
> Grant Ingersoll
> Center for Natural Language Processing
> http://www.cnlp.org/tech/lucene.asp
>
> Read the Lucene Java FAQ at http://wiki.apache.org/lucene-java/LuceneFAQ
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

------=_Part_38297_20506843.1185312516806--