lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ian Lea <ian....@gmail.com>
Subject Re: Slow Index Writes
Date Tue, 07 Jan 2014 14:51:29 GMT
I don't have a working example but I believe it's pretty
straightforward.  See  DirectoryReader.open() and .openIfChanged().


--
Ian.


On Tue, Jan 7, 2014 at 1:41 PM, Klaus Schaefers
<klaus.schaefers@ligatus.com> wrote:
> Hi,
>
>
> I was looking for some examples but I just found some using an NRTManager
> class? In Lucene 4.5 I cannot find the class (missing a maven dependency?).
> Can anyone point me to a working example?
>
> Cheers,
>
> Klaus
>
>
>
> On Fri, Jan 3, 2014 at 11:49 AM, Ian Lea <ian.lea@gmail.com> wrote:
>
>> You will indeed get poor performance if you commit for every doc.  Can
>> you compromise and commit every, say, 1000 docs, or once every few
>> minutes, or whatever makes sense for your app.
>>
>> Or look at lucene's near-real-time search features.  Google "Lucene
>> NRT" for info.
>>
>> Or use Elastic Search.
>>
>>
>> --
>> Ian.
>>
>>
>> On Fri, Jan 3, 2014 at 10:21 AM, Klaus Schaefers
>> <klaus.schaefers@ligatus.com> wrote:
>> > Hi,
>> >
>> > I am trying to use a lucene as a kind of key value store, but I
>> encountered
>> > some bad performance issues. When I try to add my data as documents to
>> the
>> > index I get an average write rate of 3 documents / second!! This seems to
>> > me ridiculously slow and I guess I must have somewhere an error. Please
>> > have a look at my code:
>> >
>> >
>> >
>> >         Directory dir = new NIOFSDirectojava-user@lucene.apache.org!
>> > java-user@lucene.apache.org!ry(file);
>> >         Analyzer analyzer =  new StandardAnalyzer(Version.LUCENE_45);
>> >         IndexWriterConfig config = new
>> IndexWriterConfig(Version.LUCENE_45,
>> > analyzer);
>> >         IndexWriter writer = new IndexWriter(dir, config);
>> >
>> >         int eventCount = 1000;
>> >         for(int i=0; i < eventCount;i++){
>> >             Document doc = new Document();
>> >             doc.add(new StringField("id", i+"id" ,Store.YES));
>> >             doc.add(new StoredField("b", buildVector()));
>> >             writer.addDocument(doc);
>> >             writer.commit();
>> >         }
>> >         dir.close();
>> >         writer.close()
>> >
>> >
>> > Not calling the commit function seems to fix the issue, but I guess this
>> > would then have some issues if I want to read values in the mean time. My
>> > normal use case would be to read something from the index, maybe alter it
>> > and then write back. So I would have roughly 50% of reads.
>> >
>> > I tried also an embedded version of elastic search and it manages to go
>> to
>> > 2000 documents/ per second. As its based on lucene as well I guess I do
>> > something wrong in my code.
>> >
>> >
>> > THX for the help,
>> >
>> > Klaus
>> >
>> >
>> > --
>> >
>> > --
>> >
>> > Klaus Schaefers
>> > Senior Optimization Manager
>> >
>> > Ligatus GmbH
>> > Hohenstaufenring 30-32
>> > D-50674 Köln
>> >
>> > Tel.:  +49 (0) 221 / 56939 -784
>> > Fax:  +49 (0) 221 / 56 939 - 599
>> > E-Mail: klaus.schaefers@ligatus.com
>> > Web: www.ligatus.de
>> >
>> > HRB Köln 56003
>> > Geschäftsführung:
>> > Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
>> > Dipl.-Wirtschaftsingenieur Arne Wolter
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
>
> --
>
> --
>
> Klaus Schaefers
> Senior Optimization Manager
>
> Ligatus GmbH
> Hohenstaufenring 30-32
> D-50674 Köln
>
> Tel.:  +49 (0) 221 / 56939 -784
> Fax:  +49 (0) 221 / 56 939 - 599
> E-Mail: klaus.schaefers@ligatus.com
> Web: www.ligatus.de
>
> HRB Köln 56003
> Geschäftsführung:
> Dipl.-Kaufmann Lars Hasselbach, Dipl.-Kaufmann Klaus Ludemann,
> Dipl.-Wirtschaftsingenieur Arne Wolter

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message