lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chew Yee Chuang" <yeechu...@tecforte.com>
Subject RE: High CPU usage duing index and search
Date Wed, 01 Aug 2007 02:29:30 GMT
Hi,

Thanks for the link provided, actually I've go through those article when I
developing the index and search function for my application. I haven’t try
profiler yet, but I monitor the CPU usage and notice that whatever index or
search performing, the CPU usage raise to 100%. Below I will try to
elaborate more on what my application is doing and how I index and search.

There are many concurrent process running, first, the application will write
records that received into a text file with tab separated each different
field. Application will point to a new file every 10mins and start writing
to it. So every file will contains only 10mins record, approximate 600,000
records per file. Then, the indexing process will check whether there is a
text file to be index, if it is, the thread will wake up and start perform
indexing.
 
The indexing process will first add documents to RAMDir, Then later, add
RAMDir into FSDir by calling addIndexNoOptimize() when there is 100,000
documents(32 fields per doc) in RAMDir. There is only 1 IndexWriter(FSDir)
was created but a few IndexWriter(RAMDir) was created during the whole
process. Below are some configuration for IndexWriters that I mentioned:-

IndexWriter (RAMDir)
- SimpleAnalyzer
- setMaxBufferedDocs(10000)
- Filed.Store.YES
- Field.Index.NO_NORMS

IndexWriter (FSDir)
- SimpleAnalyzer
- setMergeFactor(20)
- addIndexesNoOptimize()

For the searching, because there are many queries(20,000) run continuously
to generate the aggregate table for reporting purpose. All this queries is
run in nested loop, and there is only 1 Searcher created, I try searcher and
filter as well, filter give me better result, but both also utilize lots of
CPU resources.

Hope this info will help and sorry for my bad English.

Thanks
eChuang, Chew

-----Original Message-----
From: karl wettin [mailto:karl.wettin@gmail.com] 
Sent: Tuesday, July 31, 2007 5:54 PM
To: java-user@lucene.apache.org
Subject: Re: High CPU usage duing index and search


31 jul 2007 kl. 05.25 skrev Chew Yee Chuang:
> But just notice that when Lucene performing search or index,
> the CPU usage on my machine raise to 100%, because of this issue,  
> some of my
> others backend process will slow down eventually. Just want to know  
> does
> anyone face this problem before ? and is it any idea on how to  
> overcome this
> problem ?

Did you run a profiler to see what it is that consume all the resources?
It is very hard to guess based on the information you supplied. Start  
here:

http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
http://wiki.apache.org/lucene-java/ImproveSearchingSpeed


-- 
karl

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


No virus found in this incoming message.
Checked by AVG Free Edition. 
Version: 7.5.476 / Virus Database: 269.11.0/927 - Release Date: 7/30/2007
5:02 PM
 

No virus found in this outgoing message.
Checked by AVG Free Edition. 
Version: 7.5.476 / Virus Database: 269.11.0/929 - Release Date: 7/31/2007
5:26 PM
 



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message