Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of cutting@gmail.com designates
 66.249.82.236 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding:sender;
        b=cTLNa/0xiYv4tDcPEF/4igTeqK5Z7f5N9z4YhOwlJtBUtZGuvhRU/OOh6qMwyYgcdP
         YUjRc6sNRXaCSCADMWd982xn6U1ysE5MGbN1QVZnRwDpPjg2ZyVVRWLhubd9Bfz0WR8G
         QOlx/0cMoelhNHtU7Jb13jPBfnyRcjl2EP0gI=
Message-ID: <485A933B.102@apache.org>
Date: Thu, 19 Jun 2008 10:11:23 -0700
From: Doug Cutting <cutting@apache.org>
User-Agent: Thunderbird 2.0.0.14 (X11/20080505)
MIME-Version: 1.0
To: core-dev@hadoop.apache.org
Subject: Re: Gigablast.com search engine- 10BILLION PAGES!
References: <499802fc0806051220j39709729ue336405639eb0bb9@mail.gmail.com>
	 <485A5E6B.8060908@cs.put.poznan.pl>
 <c7d45fc70806191001p49ccdc38n474c6a9011b42cb2@mail.gmail.com>
In-Reply-To: <c7d45fc70806191001p49ccdc38n474c6a9011b42cb2@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Sender: Doug Cutting <cutting@gmail.com>

Ted Dunning wrote:
> One way that this sort of statement can come out of a marketing person's
> mouth is if you scan 10 billion pages, decide that 95% of them will never
> appear on any results list and only actually index 500 million.

The classic way to boost your count by an order of magnitude is to 
counts a page as "indexed" if you've only indexed an anchor to it, but 
not actually downloaded and indexed the content of the page.

Doug