lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jake Mannix" <jake.man...@gmail.com>
Subject Re: Indexing Speed: 2.3 vs 2.2 (real world numbers)
Date Mon, 04 Feb 2008 00:31:59 GMT
Yeah, I should have mentioned - this was merely with a jar replacement, we
haven't gotten around to doing fun 2.3-related stuff like making sure our
domain-specific tokenizers use the next(Token), as well as making sure set
all of our buffersizes by RAM used.

We tried multithreading the process, as we have a multi-core, multi-disk
architecture, but for some reason we never saw more than 99% (of one core)
cpu usage during indexing, as if some internal synchronization was getting
hit... I should try it again through the profiler and see if I can pinpoint
where it was getting tripped up.   On the other hand, I'm not sure if we
*need* faster than 26 minute indexing, so once we're sure we can move up to
2.3 for production, that may just solve our indexing perf issues.

Now if I can just figure out how to speed up our query performance too, I'll
be in an even *better* mood. :)

  -jake

On Feb 3, 2008 2:11 PM, Michael McCandless <lucene@mikemccandless.com>
wrote:

>
> Awesome!  We are glad to hear that :)
>
> You might be able to make it even faster with the steps here:
>
>     http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
>
> Mike
>
> Jake Mannix wrote:
>
> > Hello all,
> >   I know you lucene devs did a lot of work on indexing performance
> > in 2.3,
> > and I just tested it out last thursday, so I thought I'd let you
> > know how it
> > fared:
> >
> >   On a 2.17 million document index, a recent test gave indexing
> > time to be:
> >
> >     * lucene 2.2: 4.83 hours
> >     * lucene 2.3: 26 minutes
> >
> >   About a factor of 11 speedup.  Holy smokes!  Great work folks.
> >
> >
> >   -jake
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message