Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of jake.mannix@gmail.com
 designates 209.85.211.183 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=Sl6wPmYPNaC7kDxlY/TWaEmYE5ZHnnvkyak8CEUPie0RJn/3GsaSTe+2r21Io91BM7
         IforXwkG/gDKIfNjS9aMOcofRp4/8YL8/SnRJEgYgm0onbFhe0GGbdMKyG8+nhtZHl4m
         4mIQoCpzn+HhXuyRgJXaC33okIvkc1GmIdbqM=
MIME-Version: 1.0
In-Reply-To: 
 <E3137F97B0A0804194F369F4E7C3B71929AE7C8F75@EXCHANGE.persistent.co.in>
References: 
 <E3137F97B0A0804194F369F4E7C3B71929AE7C8F75@EXCHANGE.persistent.co.in>
Date: Thu, 22 Oct 2009 22:39:14 -0700
Message-ID: <4b124c310910222239k7d1de1c1q8cb7c54165ad2f89@mail.gmail.com>
Subject: Re: Maximum index file size
From: Jake Mannix <jake.mannix@gmail.com>
To: java-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=00163630f949e1096c047693a142

--00163630f949e1096c047693a142
Content-Type: text/plain; charset=ISO-8859-1

On Thu, Oct 22, 2009 at 10:29 PM, Hrishikesh Agashe <
hrishikesh_agashe@persistent.co.in> wrote:

> Can I create an index file with very large size, like 1 TB or so? Is there
> any limit on how large index file one can create? Also, will I be able to
> search on this 1 TB index file at all?
>

Leaving aside the question of hardware or JVM limits on monstrous files,
this question (can you search this file) is easier: if you've got say, a ten
billion documents in one index, and you have a query which is going to hit
maybe even just 0.1% of the documents, you'll need to do scoring of 10
million hits in the course of that query.  To do this in under a second
means you only have 100 nanoseconds to look at each document.  If your query
hits 1% of your documents, you're down to 10 ns per document.  I've never
tried searching a 1TB index, but I'd say that's pushing it.

Is there a reason you can't shard your index, and instead put maybe 20
shards of 50GB (or better - 100 shards of 10GB) each on a variety of
machines, and just merge results?

  -jake

--00163630f949e1096c047693a142--