Mailing-List: contact mahout-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mahout-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of ted.dunning@gmail.com
 designates 209.85.160.46 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=ENQwLzadCppe3i101P2YOYd2fv8tFgw0JW/KsNZBOKzvP4YvBUu2YrT7X6siyzoi5P
         dV9FKBV6YXRKwIZWk0Ea3z66ph6DM6XCs7UersYAiRYatQT0vN9tIO6SBzxLVgpO4m6U
         G9QVeZGSxN6JhTzSxK9SeSyUP/I9pYEcWcDEU=
MIME-Version: 1.0
In-Reply-To: <916DDB7F-CE01-469B-B027-D739B1D4A617@apache.org>
References: <5eb9b7ae0911060506n3b60dbfdmd34e41fc3db95c45@mail.gmail.com>
	<916DDB7F-CE01-469B-B027-D739B1D4A617@apache.org>
From: Ted Dunning <ted.dunning@gmail.com>
Date: Fri, 6 Nov 2009 11:57:27 -0800
Message-ID: <c7d45fc70911061157v719aaf1dwbf538a1dd744240a@mail.gmail.com>
Subject: Re: TU Berlin Winter of Code Project
To: mahout-user@lucene.apache.org
Content-Type: multipart/alternative; boundary=00504502ead504195d0477b942d2

--00504502ead504195d0477b942d2
Content-Type: text/plain; charset=UTF-8

The question that I don't see addressed is whether you choose to use a fully
streaming approach as is done in Bixo or whether you will use a document
repository approach as is more common in most search engines.

Hbase is reputedly ready enough to serve as a document repository.  Using
such an approach would be very helpful for the incremental nature of web
crawls.

What is the plan in this regard?

On Fri, Nov 6, 2009 at 11:47 AM, Grant Ingersoll <gsingers@apache.org>wrote:

>
> This is obviously only a first draft of what we think would be a suited
> overall
> architecture


-- 
Ted Dunning, CTO
DeepDyve

--00504502ead504195d0477b942d2--