Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of erickerickson@gmail.com
 designates 209.85.223.186 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=uSVQHEKLT6TlNTnSPd1Vu8yKM76UHTyr3jyleI8KmRUzkjd4cOgRklQZFGqgLt+goY
         QKit73Nbd9rKd0giFgtePpQs38DO7AAHuObzZkwbINsmw4hUDO/btX0QWw88DE3VtPgt
         6trZVAnq50O87iL6F3cAgxoPU18R9bLnqWO8E=
MIME-Version: 1.0
In-Reply-To: <1253805447.9486.30.camel@bohr>
References: <1253805447.9486.30.camel@bohr>
Date: Thu, 24 Sep 2009 12:29:12 -0400
Message-ID: <359a92830909240929v6ba2d83cn6f888c89d1c12c7a@mail.gmail.com>
Subject: Re: metrics for index ~100M docs
From: Erick Erickson <erickerickson@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=0016e645b8cae5a21804745554cf

--0016e645b8cae5a21804745554cf
Content-Type: text/plain; charset=ISO-8859-1

It's really hard to say anything meaningful here. How many fields? Whatkind
of sorting to you intend to do? How complex are the queries you
expect?

And even if you have meaningful answers to the above,
then "it depends" (tm).

Then you could go to SOLR (which is built on Lucene) to handle
distributed searching and a host of other infrastructure issues.

There are certainly Lucene installations out there that are much larger
than you're considering if that helps.

But you can create a small test app *very* quickly that'll help you
answer this for your local set of conditions, which might be a good
place to start.

Don't forget the "powered by" section of the Wiki for some ideas:
http://wiki.apache.org/lucene-java/PoweredBy

<http://wiki.apache.org/lucene-java/PoweredBy>Best
Erick

On Thu, Sep 24, 2009 at 11:17 AM, Joel Halbert <joel@su3analytics.com>wrote:

> Hi,
>
> Does anyone know of any recent metrics & stats on building out an index
> of  ~100mm documents (each doc approx 5k). I'm looking for approx stats
> on time to build, time to query and infrastructure requirements (number
> of machines & spec) to reasonably support an index of such a size.
>
> Thanks,
> Joel
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

--0016e645b8cae5a21804745554cf--