Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of jake.mannix@gmail.com
 designates 72.14.246.249 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=uCpSfRMmbQl/xUOjmza8KRp26gX0Dyxmxj81lvYjwWKRrJr7aKpJAATzalmToEJLAZIby/kQiS6fXOnBQFo1sh3ADxzVcRkV/CL2vDQK0rWJXDarq3DbhbuKrT9t2AxU9D8JB1qMtsIzr+UNlk+jRCzEF+HLOkWj1QnOSnGwyG0=
Message-ID: <4b124c310802031631x11fc3d1ah82f6d6b506b1e4b5@mail.gmail.com>
Date: Sun, 3 Feb 2008 16:31:59 -0800
From: "Jake Mannix" <jake.mannix@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Indexing Speed: 2.3 vs 2.2 (real world numbers)
In-Reply-To: <E142FB18-3A38-489F-90AA-BB6F0AB2C9B7@mikemccandless.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_3131_6914601.1202085119461"
References: <4b124c310802031157l2e7790b9oe1608beb6c5a6dd6@mail.gmail.com>
	 <E142FB18-3A38-489F-90AA-BB6F0AB2C9B7@mikemccandless.com>

------=_Part_3131_6914601.1202085119461
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Yeah, I should have mentioned - this was merely with a jar replacement, we
haven't gotten around to doing fun 2.3-related stuff like making sure our
domain-specific tokenizers use the next(Token), as well as making sure set
all of our buffersizes by RAM used.

We tried multithreading the process, as we have a multi-core, multi-disk
architecture, but for some reason we never saw more than 99% (of one core)
cpu usage during indexing, as if some internal synchronization was getting
hit... I should try it again through the profiler and see if I can pinpoint
where it was getting tripped up.   On the other hand, I'm not sure if we
*need* faster than 26 minute indexing, so once we're sure we can move up to
2.3 for production, that may just solve our indexing perf issues.

Now if I can just figure out how to speed up our query performance too, I'll
be in an even *better* mood. :)

  -jake

On Feb 3, 2008 2:11 PM, Michael McCandless <lucene@mikemccandless.com>
wrote:

>
> Awesome!  We are glad to hear that :)
>
> You might be able to make it even faster with the steps here:
>
>     http://wiki.apache.org/lucene-java/ImproveIndexingSpeed
>
> Mike
>
> Jake Mannix wrote:
>
> > Hello all,
> >   I know you lucene devs did a lot of work on indexing performance
> > in 2.3,
> > and I just tested it out last thursday, so I thought I'd let you
> > know how it
> > fared:
> >
> >   On a 2.17 million document index, a recent test gave indexing
> > time to be:
> >
> >     * lucene 2.2: 4.83 hours
> >     * lucene 2.3: 26 minutes
> >
> >   About a factor of 11 speedup.  Holy smokes!  Great work folks.
> >
> >
> >   -jake
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

------=_Part_3131_6914601.1202085119461--