Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Date: Fri, 07 Mar 2008 15:17:29 +0100
From: Toke Eskildsen <te@statsbiblioteket.dk>
Subject: RE: Swapping between indexes
In-reply-to: <068BD1F02ABA4E999506EF880CA98F70@msrvcn04>
To: java-user@lucene.apache.org
Reply-to: te@statsbiblioteket.dk
Message-id: <1204899449.9126.219.camel@PC990.sb.statsbiblioteket.dk>
Organization: Statsbiblioteket
MIME-version: 1.0
Content-type: text/plain
Content-transfer-encoding: 7BIT
References: <227621ad0803060230i485556av7b58bcd6bfbba6cc@mail.gmail.com>
 <3A267701-C68A-4A99-88D2-3220D6237F0C@mikemccandless.com>
 <227621ad0803060502j11adff3dg547fc7dbb4da1526@mail.gmail.com>
 <c68e39170803060657q59e34990w772c1a309c275cca@mail.gmail.com>
 <F6B77DC74E994492B9F89B665911EE65@msrvcn04>
 <c68e39170803060930i79d8faf0m6dd729ac4d65fe4b@mail.gmail.com>
 <068BD1F02ABA4E999506EF880CA98F70@msrvcn04>

On Thu, 2008-03-06 at 18:40 +0100, spring@gmx.eu wrote:
> > >  With a commit after every add: 30 min.
> > >  With a commit after 100 add: 23 min.
> > >  Only one commit: 20 min.

[...]

> I think it is a real world scenario because one has always the read the docs
> from somewhere and offen has to store the index state somewhere else.

Very true, but the time it takes to create the documents varies greatly
between systems.

I tried repeating your test by creating a simple 14 MB index with 10,000
documents on my desktop-machine. each document was made up of

 - one non-tokenized unique stored indexed field
 - one non-tokenized indexed stored field with one of 9 terms
 - one tokenized field with 930 random characters, including space

With a commit after every add: 4 min, 46 sec.
With a commit after every 100 add: 12 sec.
Only one commit: 8 sec.


Guesstimating the amortized time spend on adding each document on such a
small corpus, by blatantly ignoring the overhead of creating the
documents, gives us the following:

With a commit after every add: (286 sec / 10,000 docs) 28.6 ms.
With a commit after every 100 add: (12 sec / 10,000 docs) 1.2 ms.
Only one commit: (8 sec / 10,000 docs) 0.8 ms.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org