Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of tsuraan@gmail.com designates
 209.85.221.203 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=ELAtgHw4jyn96KCz/4/3SDvAVI/Z4VYq8gCkI6aLsbkXER7O2rq1Ni9mb52A+MOvDM
         WbnWS0w+SQuFcQo7yi3YvRz6fFoSwepEi1wVx2J9JijLeft7qAo0ITVgoIZM5LgwtwBL
         hzVuyy8QXsxpYtjvAHYFxH5swAfSc6i7gSAqQ=
MIME-Version: 1.0
In-Reply-To: <4A6746F1.7080906@informatics.jax.org>
References: <84fb38e30907220959n40398f04rab899b3bb4ad5a18@mail.gmail.com>
	 <4A6746F1.7080906@informatics.jax.org>
Date: Wed, 22 Jul 2009 12:41:39 -0500
Message-ID: <84fb38e30907221041o2c02ed7co686eb0d24e1dceb3@mail.gmail.com>
Subject: Re: Batch searching
From: tsuraan <tsuraan@gmail.com>
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

> If you did this, wouldn't you be binding the processing of the results
> of all queries to that of the slowest performing one within the collection?

I would imagine it would, but I haven't seen too much variance between
lucene query speeds in our data.

> I'm guessing you are trying for some sort of performance benefit by
> batch processing, but I question whether or not you will actually get
> more performance by performing your queries in a threaded type
> environment, and then processing their results as they come in.
>
> Could you give a bit more description about what you are actually trying
> to accomplish, I'm sure this list could help better if we had more
> information.

What I'd like to do is build lots of small indices (a few thousand
documents per index) and put them into HDFS for search distribution.
We already have our own map-reduce framework for searching, but HDFS
seems to be a really good fit for an actual storage mechanism.

My concern is that when we have one searcher using thousands of
HDFS-backed indices, the seeking might get a bit nasty. HDFS
apparently has pretty good seeking performance, but it really looks
like it was designed for streaming, so if I could make my searches use
sequential index access, I would expect better performance than having
a ton of simultaneous searches making HDFS seek all over the place.

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org