lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From testn <te...@doramail.com>
Subject RE: High CPU usage duing index and search
Date Thu, 02 Aug 2007 11:10:06 GMT

20,000 queries continuously? Sounds a bit too much. Can you elaborate more
what you need to do? Probably you won't need that many queries.



Chew Yee Chuang wrote:
> 
> Hi,
> 
> Thanks for the link provided, actually I've go through those article when
> I
> developing the index and search function for my application. I haven’t try
> profiler yet, but I monitor the CPU usage and notice that whatever index
> or
> search performing, the CPU usage raise to 100%. Below I will try to
> elaborate more on what my application is doing and how I index and search.
> 
> There are many concurrent process running, first, the application will
> write
> records that received into a text file with tab separated each different
> field. Application will point to a new file every 10mins and start writing
> to it. So every file will contains only 10mins record, approximate 600,000
> records per file. Then, the indexing process will check whether there is a
> text file to be index, if it is, the thread will wake up and start perform
> indexing.
>  
> The indexing process will first add documents to RAMDir, Then later, add
> RAMDir into FSDir by calling addIndexNoOptimize() when there is 100,000
> documents(32 fields per doc) in RAMDir. There is only 1 IndexWriter(FSDir)
> was created but a few IndexWriter(RAMDir) was created during the whole
> process. Below are some configuration for IndexWriters that I mentioned:-
> 
> IndexWriter (RAMDir)
> - SimpleAnalyzer
> - setMaxBufferedDocs(10000)
> - Filed.Store.YES
> - Field.Index.NO_NORMS
> 
> IndexWriter (FSDir)
> - SimpleAnalyzer
> - setMergeFactor(20)
> - addIndexesNoOptimize()
> 
> For the searching, because there are many queries(20,000) run continuously
> to generate the aggregate table for reporting purpose. All this queries is
> run in nested loop, and there is only 1 Searcher created, I try searcher
> and
> filter as well, filter give me better result, but both also utilize lots
> of
> CPU resources.
> 
> Hope this info will help and sorry for my bad English.
> 
> Thanks
> eChuang, Chew
> 
> -----Original Message-----
> From: karl wettin [mailto:karl.wettin@gmail.com] 
> Sent: Tuesday, July 31, 2007 5:54 PM
> To: java-user@lucene.apache.org
> Subject: Re: High CPU usage duing index and search
> 
> 
> 31 jul 2007 kl. 05.25 skrev Chew Yee Chuang:
>> But just notice that when Lucene performing search or index,
>> the CPU usage on my machine raise to 100%, because of this issue,  
>> some of my
>> others backend process will slow down eventually. Just want to know  
>> does
>> anyone face this problem before ? and is it any idea on how to  
>> overcome this
>> problem ?
> 
> Did you run a profiler to see what it is that consume all the resources?
> It is very hard to guess based on the information you supplied. Start  
> here:
> 
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
> http://wiki.apache.org/lucene-java/ImproveSearchingSpeed
> 
> 
> -- 
> karl
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> No virus found in this incoming message.
> Checked by AVG Free Edition. 
> Version: 7.5.476 / Virus Database: 269.11.0/927 - Release Date: 7/30/2007
> 5:02 PM
>  
> 
> No virus found in this outgoing message.
> Checked by AVG Free Edition. 
> Version: 7.5.476 / Virus Database: 269.11.0/929 - Release Date: 7/31/2007
> 5:26 PM
>  
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/High-CPU-usage-duing-index-and-search-tf4190756.html#a11962524
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message