Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of john.wang@gmail.com
 designates 209.85.128.191 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references;
        b=QaVICEAn6HfBPzZTzC5eIhhJF1DLub/QFvQFOsB/D9GE69WsQdDee/fNWSLvW0Ua3dDqo8vYn6RAC36ICMtgqgPo2sEEh/l2lsZPpqJ+Xm2yg8g2MBpCVHsLgCCNoiw5/XxsIOAM69wnlnWsP9c5+Na9hO3BH2HplqSR4Ztof7Y=
Message-ID: <8837fb770805011310p5fb7b1e7wcc75ab9992fed6fe@mail.gmail.com>
Date: Thu, 1 May 2008 13:10:39 -0700
From: "John Wang" <john.wang@gmail.com>
To: java-user@lucene.apache.org
Subject: Re: Does Lucene Supports Billions of data
In-Reply-To: <200805010910.04480.daniel@nuix.com>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_7011_29403196.1209672639913"
References: <16974808.post@talk.nabble.com>
	 <5e76f3840804300646y7d230aa3xb7827193fe441b4d@mail.gmail.com>
	 <8837fb770804300701v74791ce0h6c1bf359b801e473@mail.gmail.com>
	 <200805010910.04480.daniel@nuix.com>

------=_Part_7011_29403196.1209672639913
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

I am not sure why this is the case, docid is internal to the sub index. As
long as the sub index size is below 2 bil, there is no need for docid to be
long. With multiple indexes, I was thinking having an aggregater which
merges maybe only a page of search result.

Example:

sub index 1: 1 billion docs
sub index 2: 1 billion docs
sub index 3: 1 billion docs

federating search to these subindexes, you represent an index of 3 billion
docs, and all internal doc ids are of type int.

Maybe I am not understanding something.

-John

On Wed, Apr 30, 2008 at 4:10 PM, Daniel Noll <daniel@nuix.com> wrote:

> On Thursday 01 May 2008 00:01:48 John Wang wrote:
> > I am not sure how well lucene would perform with > 2 Billion docs in a
> > single index anyway.
>
> Even if they're in multiple indexes, the doc IDs being ints will still
> prevent
> it going past 2Gi unless you wrap your own framework around it.
>
> Daniel
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

------=_Part_7011_29403196.1209672639913--