Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (herse.apache.org: local policy)
Date: Thu, 1 Mar 2007 10:28:07 +0200
From: "Nadav Har'El" <nyh@math.technion.ac.il>
To: java-user@lucene.apache.org
Subject: Re: indexing performance
Message-ID: <20070301082807.GA15826@fermat.math.technion.ac.il>
References: <f514763f0702270622x2770d82cq64610fe7e9ec90fd@mail.gmail.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <f514763f0702270622x2770d82cq64610fe7e9ec90fd@mail.gmail.com>
User-Agent: Mutt/1.4.2.2i
Hebrew-Date: 11 Adar 5767

On Tue, Feb 27, 2007, Saravana wrote about "indexing performance":
> Hi,
> 
> Is it possible to scale lucene indexing like 2000/3000 documents per
> second?

I don't know about the actual numbers, but one trick I've used in the past
to get really fast indexing was to create several independent indexes in
parallel. Simply, if you have, say, 4 CPUs and perhaps even several physical
disks, run 4 indexing processes each indexing a 1/4 of the files and creating
a separate index (on separate disks on separate IO channels, if possible).

At the end, you have 4 indexes which you can actually search together without
any real need to merge them, unless query performance is very important to
you as well.

> I need to index 10 fields each with 20 bytes long.  I should be
> able to search by just giving any of the field values as criteria. I need to
> get the count that has same field values.

You need just the counts? And you want to do just whole-field matching, not
word matching? In that case, Lucene might be an overkill for you. Or, if you
do use Lucene, make sure to use "keyword" (untokenized) fields, not
"tokenized" fields.

-- 
Nadav Har'El                        |      Thursday, Mar  1 2007, 11 Adar 5767
IBM Haifa Research Lab              |-----------------------------------------
                                    |Open your arms to change, but don't let
http://nadav.harel.org.il           |go of your values.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org