Mailing-List: contact general-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: general@lucene.apache.org
Received-SPF: pass (asf.osuosl.org: domain of yseeley@gmail.com designates
 66.249.92.171 as permitted sender)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
        s=beta; d=gmail.com;
        h=received:message-id:date:from:sender:to:subject:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth;
        b=LbvAMZCJRdT7n3GDhzBQn5VnN2R95Vl8Rh0UkoEKjNYWsPY9YnRys2vMlzIJS1hU3SyV8oFbY0NPLi/+3P1UPpNUk/6gNes1/d6BZPympS40frA7fGnIZ754zV8EblgpwKh+y38kw236CtafP1/HHX1hR0OA42rnJ1Brv+OsDXo=
Message-ID: <c68e39170610191823g6814b8a7p1f7458f607b17115@mail.gmail.com>
Date: Thu, 19 Oct 2006 21:23:58 -0400
From: "Yonik Seeley" <yonik@apache.org>
Sender: yseeley@gmail.com
To: general@lucene.apache.org
Subject: Re: [PROPOSAL] index server project
In-Reply-To: 
 <392521EA2692A2418DF48C331E61E32522D3@professorville.windows.esseff.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
References: 
 <392521EA2692A2418DF48C331E61E32522D3@professorville.windows.esseff.org>

On 10/19/06, Steven Parkes <steven_parkes@esseff.org> wrote:
> You mention partitioning of indexes, though mostly around delete. What
> about scalability of corpus size?

Definitely in scope.  Solr already has scalability of search volume
via searchers behind of a load balancer all getting their index from a
master.  The problem comes when an index is too big to get decent
latency for a single query, and that's when you need to partiton the
index into "shards" to use google terminology.

> Would partitioning be effective for
> that, too?

Yes, to a certain extent.  At some point you run into network
bandwidth issues if you go deep into rankings.

> What about scalability of ingest rate?

As it relates to indexing, I think nutch already has that base covered.

> What are you thinking, in terms of size? Is this a 10 node thing?

I'm personally interested in perhaps 10 to 20 index shards, with
multiple replicas of each shard for HA and query load scalability.

> A 1000
> node thing? More? Bigger is cool, but raises a lot of issues.

Should be possible, but I won't personally be looking for that.  I
think scaling effectively will be partially in the hands of the client
and how it chooses to merge results from shards.

> How
> dynamic?

> Can nodes come and go?

Unplanned: yes.  HA is personally key for me.
Planned (adding capacity gracefully): it would be nice.  I actually
hadn't planned it for Solr.

> Are you going to assume homogeneity of
> nodes?

Hardware homogeneity?  That might be out of scope... I'd start off
without worrying about it in any case.

> What about add/modify/delete to search visibility latency? Close to
> batch/once-a-day or real-time?

Anywhere in between I'd think.  "Realtime" latencies of minutes or
longer are normally fine.

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server