Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of dansegel@gmail.com designates
 209.85.146.183 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:mime-version:content-type;
        b=HFHqo52lgh/tWDVLRKm6+n9sfHkGs5bHMvWuRzmN6VsRKPzVSW14FWdUkuqWXRVf/u
         Lwtj6VE3YgBYbaDTM+sodr4fGaO75IkYxxhLmHBP8T/00MRyaIPlhpkgFMbljWoiJMQ8
         5dN8fZuX7+fu9b36qpzwQnpMaBECvS5+XJMyM=
Message-ID: <499802fc0806050612u1e711e21u466887582767dd7d@mail.gmail.com>
Date: Thu, 5 Jun 2008 09:12:31 -0400
From: "Dan Segel" <dansegel@gmail.com>
To: core-user@hadoop.apache.org
Subject: Gigablast.com search engine, 10billion pages!!!
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_1595_1833856.1212671551357"

------=_Part_1595_1833856.1212671551357
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Our ultimate goal is to basically replicate gigablast.com search engine.
They claim to have less than 500 servers that contain 10billion pages
indexed, spidered and updated on a routine basis...  I am looking at
featuring 500 million pages indexed per node, and have a total of 20 nodes.
Each node will feature 2 quad core processes, 4TB (at raid 5) and 32 gb of
ram.  I believe this can be done however how many searches per second do you
think would be realistic in this instance?  We are looking at achieving
25+/- searches per second ultimately spread out over the 20 nodes... I can
really uses some advice with this one.

Thanks,
D. Segel

------=_Part_1595_1833856.1212671551357--