lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Slow Index Writes
Date Fri, 03 Jan 2014 10:49:29 GMT
You will indeed get poor performance if you commit for every doc.  Can
you compromise and commit every, say, 1000 docs, or once every few
minutes, or whatever makes sense for your app.

Or look at lucene's near-real-time search features.  Google "Lucene
NRT" for info.

Or use Elastic Search.


--
Ian.


On Fri, Jan 3, 2014 at 10:21 AM, Klaus Schaefers
<klaus.schaefers@ligatus.com> wrote:
> Hi,
>
> I am trying to use a lucene as a kind of key value store, but I encountered
> some bad performance issues. When I try to add my data as documents to the
> index I get an average write rate of 3 documents / second!! This seems to
> me ridiculously slow and I guess I must have somewhere an error. Please
> have a look at my code:
>
>
>
>         Directory dir = new NIOFSDirectojava-user@lucene.apache.org!
> java-user@lucene.apache.org!ry(file);
>         Analyzer analyzer =  new StandardAnalyzer(Version.LUCENE_45);
>         IndexWriterConfig config = new IndexWriterConfig(Version.LUCENE_45,
> analyzer);
>         IndexWriter writer = new IndexWriter(dir, config);
>
>         int eventCount = 1000;
>         for(int i=0; i < eventCount;i++){
>             Document doc = new Document();
>             doc.add(new StringField("id", i+"id" ,Store.YES));
>             doc.add(new StoredField("b", buildVector()));
>             writer.addDocument(doc);
>             writer.commit();
>         }
>         dir.close();
>         writer.close()
>
>
> Not calling the commit function seems to fix the issue, but I guess this
> would then have some issues if I want to read values in the mean time. My
> normal use case would be to read something from the index, maybe alter it
> and then write back. So I would have roughly 50% of reads.
>
> I tried also an embedded version of elastic search and it manages to go to
> 2000 documents/ per second. As its based on lucene as well I guess I do
> something wrong in my code.
>
>
> THX for the help,
>
> Klaus
>
>
> --
>
> --
>
> Klaus Schaefers
> Senior Optimization Manager
>
> Ligatus GmbH
> Hohenstaufenring 30-32
> D-50674 Köln
>
> Tel.:  +49 (0) 221 / 56939 -784
> Fax:  +49 (0) 221 / 56 939 - 599
> E-Mail: klaus.schaefers@ligatus.com
> Web: www.ligatus.de
>
> HRB Köln 56003
> Geschäftsführung:
> Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> Dipl.-Wirtschaftsingenieur Arne Wolter

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message