Return-Path: Delivered-To: apmail-lucene-java-user-archive@www.apache.org Received: (qmail 99933 invoked from network); 11 Aug 2005 03:55:41 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 11 Aug 2005 03:55:41 -0000 Received: (qmail 51023 invoked by uid 500); 11 Aug 2005 03:55:37 -0000 Delivered-To: apmail-lucene-java-user-archive@lucene.apache.org Received: (qmail 50982 invoked by uid 500); 11 Aug 2005 03:55:36 -0000 Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-user@lucene.apache.org Delivered-To: mailing list java-user@lucene.apache.org Received: (qmail 50965 invoked by uid 99); 11 Aug 2005 03:55:36 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Aug 2005 20:55:36 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests=RCVD_BY_IP,SPF_HELO_PASS,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (asf.osuosl.org: domain of yseeley@gmail.com designates 64.233.170.201 as permitted sender) Received: from [64.233.170.201] (HELO rproxy.gmail.com) (64.233.170.201) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 10 Aug 2005 20:55:57 -0700 Received: by rproxy.gmail.com with SMTP id g11so236699rne for ; Wed, 10 Aug 2005 20:55:33 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references; b=XDa+7vSPIg6l8rFLWKMZtkU049+c5nsCWMN7oIyVUGR61hMSJoKQwOpsR48TMraBcIsAFBqQfjHZ9dWoxFIUd5qqPMjFnnj8ecV/b4GredTGIsB3L/yPeJStb6Yu5JiGvgCuWtyKDW+/GCqu1XxTr0Tqb7JKC1eUiYjDLKdCVNA= Received: by 10.38.12.40 with SMTP id 40mr494579rnl; Wed, 10 Aug 2005 20:55:33 -0700 (PDT) Received: by 10.38.12.41 with HTTP; Wed, 10 Aug 2005 20:55:33 -0700 (PDT) Message-ID: Date: Wed, 10 Aug 2005 23:55:33 -0400 From: Yonik Seeley To: java-user@lucene.apache.org Subject: Re: max number of documents In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline References: X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N I think it would be 2 billion. There are many places that woudn't like the overflow to negative docids I think... We have indexes up to 200M documents, so 1/10th the max. 64 bit ids are definitely something to think about for the near future. > Who's got Lucene indexes nearing the maximum integer value? I'm > seriously curious. That's over 4 billion documents. >=20 > As for scaling - at that point I recommend checking out Nutch's > architecture, especially the new MapReduce branch. It's made for > this sort of big time scalability. >=20 > Erik I'm still trying to grok MapReduce... -Yonik --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org For additional commands, e-mail: java-user-help@lucene.apache.org