lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From karl wettin <karl.wet...@gmail.com>
Subject Re: High CPU usage duing index and search
Date Wed, 01 Aug 2007 23:10:41 GMT
It sounds like you have a fairly busy system, perhaps 100% load on the
process is not that strange, at least not during short periods of time.

A simpler solution would be to nice the process a little bit in order to
give your background jobs some more time to think.

Running a profiler is still the best advice I can think of. It should
clearly show you what is going on when you run out of CPU.

--  
karl

1 aug 2007 kl. 04.29 skrev Chew Yee Chuang:

> Hi,
>
> Thanks for the link provided, actually I've go through those  
> article when I
> developing the index and search function for my application. I  
> haven’t try
> profiler yet, but I monitor the CPU usage and notice that whatever  
> index or
> search performing, the CPU usage raise to 100%. Below I will try to
> elaborate more on what my application is doing and how I index and  
> search.
>
> There are many concurrent process running, first, the application  
> will write
> records that received into a text file with tab separated each  
> different
> field. Application will point to a new file every 10mins and start  
> writing
> to it. So every file will contains only 10mins record, approximate  
> 600,000
> records per file. Then, the indexing process will check whether  
> there is a
> text file to be index, if it is, the thread will wake up and start  
> perform
> indexing.
>
> The indexing process will first add documents to RAMDir, Then  
> later, add
> RAMDir into FSDir by calling addIndexNoOptimize() when there is  
> 100,000
> documents(32 fields per doc) in RAMDir. There is only 1 IndexWriter 
> (FSDir)
> was created but a few IndexWriter(RAMDir) was created during the whole
> process. Below are some configuration for IndexWriters that I  
> mentioned:-
>
> IndexWriter (RAMDir)
> - SimpleAnalyzer
> - setMaxBufferedDocs(10000)
> - Filed.Store.YES
> - Field.Index.NO_NORMS
>
> IndexWriter (FSDir)
> - SimpleAnalyzer
> - setMergeFactor(20)
> - addIndexesNoOptimize()
>
> For the searching, because there are many queries(20,000) run  
> continuously
> to generate the aggregate table for reporting purpose. All this  
> queries is
> run in nested loop, and there is only 1 Searcher created, I try  
> searcher and
> filter as well, filter give me better result, but both also utilize  
> lots of
> CPU resources.
>
> Hope this info will help and sorry for my bad English.
>
> Thanks
> eChuang, Chew
>
> -----Original Message-----
> From: karl wettin [mailto:karl.wettin@gmail.com]
> Sent: Tuesday, July 31, 2007 5:54 PM
> To: java-user@lucene.apache.org
> Subject: Re: High CPU usage duing index and search
>
>
> 31 jul 2007 kl. 05.25 skrev Chew Yee Chuang:
>> But just notice that when Lucene performing search or index,
>> the CPU usage on my machine raise to 100%, because of this issue,
>> some of my
>> others backend process will slow down eventually. Just want to know
>> does
>> anyone face this problem before ? and is it any idea on how to
>> overcome this
>> problem ?
>
> Did you run a profiler to see what it is that consume all the  
> resources?
> It is very hard to guess based on the information you supplied. Start
> here:
>
> http://wiki.apache.org/lucene-java/BasicsOfPerformance
> http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
> http://wiki.apache.org/lucene-java/ImproveSearchingSpeed
>
>
> -- 
> karl
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.476 / Virus Database: 269.11.0/927 - Release Date:  
> 7/30/2007
> 5:02 PM
>
>
> No virus found in this outgoing message.
> Checked by AVG Free Edition.
> Version: 7.5.476 / Virus Database: 269.11.0/929 - Release Date:  
> 7/31/2007
> 5:26 PM
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message