Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of jason.rutherglen@gmail.com
 designates 74.125.44.30 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:in-reply-to:mime-version
         :content-type:references;
        b=s5gGB25Nzs02DqndukxDIFwexnTedlF/hhAgnoeXe4bdZLjloFEU0v1mqJq7LMNVrf
         GbZgOYqQs867P1fHlsH9tZqfYl+ZA1l0wzMjEf6vRn4D+9WjLNL9h6Z6g6l9750i1aav
         4kKZWYhTl2CALd3usUFL2/Jolsx2CN2TfrfMs=
Message-ID: <85d3c3b60812241023u72b01463r38bb9b3d1ffd8ac@mail.gmail.com>
Date: Wed, 24 Dec 2008 10:23:48 -0800
From: "Jason Rutherglen" <jason.rutherglen@gmail.com>
To: java-dev@lucene.apache.org
Subject: Re: Realtime Search
In-Reply-To: <49527708.4090109@apache.org>
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_117151_2806189.1230143028250"
References: <85d3c3b60812231751k60f00283r95b8d65b2b7adf45@mail.gmail.com>
	 <49527708.4090109@apache.org>

------=_Part_117151_2806189.1230143028250
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

> Also, what are the requirements?  Must a document be visible to search
within 10ms of being added?

0-5ms.  Otherwise it's not realtime, it's batch indexing.  The realtime
system can support small batches by encoding them into RAMDirectories if
they are of sufficient size.

> Or must it be visible to search from the time that the call to add it
returns?

Most people probably expect the update latency offered by SQL databases.

> As a baseline, how fast is it to simply use RAMDirectory?

It depends on how fast searches over the realtime index need to be.  The
detriment to speed occurs with having many small segments that are
continuously decoded (terms, postings, etc).  The advantage of MemoryIndex
and InstantiatedIndex is an actual increase in search speed compared with
RAMDirectory (see the Performance Notes at
http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc//org/apache/lucene/index/memory/MemoryIndex.htmland
)and no need to continuously decode segments that are short lived.

Anecdotal tests indicated the merging overhead of using RAMDirectory as
compared with MI or II is significant enough to make it only useful for
doing batches in the 1000s which does not seem to be what people expect from
realtime search.

On Wed, Dec 24, 2008 at 9:53 AM, Doug Cutting <cutting@apache.org> wrote:

> Jason Rutherglen wrote:
>
>> 2) Implement realtime search by incrementally creating and merging readers
>> in memory.  The system would use MemoryIndex or InstantiatedIndex to quickly
>> (more quickly than RAMDirectory) create indexes from added documents.
>>
>
> As a baseline, how fast is it to simply use RAMDirectory?  If one, e.g.,
> flushes changes every 10ms or so, and has a background thread that uses
> IndexReader.reopen() to keep a fresh version for reading?
>
> Also, what are the requirements?  Must a document be visible to search
> within 10ms of being added?  Or must it be visible to search from the time
> that the call to add it returns?  In the latter case one might still use an
> approach like the above.  Writing a small new segment to a RAMDirectory and
> then, with no merging, calling IndexReader.reopen(), should be quite fast.
>  All merging could be done in the background, as should post-merge reopens()
> that involve large segments.
>
> In short, I wonder if new reader and writer implementations are in fact
> required or whether, perhaps with a few optimizations, the existing
> implementations might meet this need.
>
> Doug
>
> ---------------------------------------------------------------------
>
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

------=_Part_117151_2806189.1230143028250
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

&gt; Also, what are the requirements? &nbsp;Must a document be visible to search within 10ms of being added? <br><br>0-5ms.&nbsp; Otherwise it&#39;s not realtime, it&#39;s batch indexing.&nbsp; The realtime system can support small batches by encoding them into RAMDirectories if they are of sufficient size.<br>
<br>&gt; Or must it be visible to search from the time that the call to add it returns? <br><br>Most people probably expect the update latency offered by SQL databases.<br><br>&gt; As a baseline, how fast is it to simply use RAMDirectory? <br>
<br>It depends on how fast searches over the realtime index need to be.&nbsp; The detriment to speed occurs with having many small segments that are continuously decoded (terms, postings, etc).&nbsp; The advantage of MemoryIndex and InstantiatedIndex is an actual increase in search speed compared with RAMDirectory (see the Performance Notes at <a href="http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc//org/apache/lucene/index/memory/MemoryIndex.html">http://hudson.zones.apache.org/hudson/job/Lucene-trunk/javadoc//org/apache/lucene/index/memory/MemoryIndex.html</a> and )and no need to continuously decode segments that are short lived.&nbsp; <br>
<br>Anecdotal tests indicated the merging overhead of using RAMDirectory as compared with MI or II is significant enough to make it only useful for doing batches in the 1000s which does not seem to be what people expect from realtime search.&nbsp; <br>
<br><div class="gmail_quote">On Wed, Dec 24, 2008 at 9:53 AM, Doug Cutting <span dir="ltr">&lt;<a href="mailto:cutting@apache.org">cutting@apache.org</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d">Jason Rutherglen wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
2) Implement realtime search by incrementally creating and merging readers in memory. &nbsp;The system would use MemoryIndex or InstantiatedIndex to quickly (more quickly than RAMDirectory) create indexes from added documents.<br>

</blockquote>
<br></div>
As a baseline, how fast is it to simply use RAMDirectory? &nbsp;If one, e.g., flushes changes every 10ms or so, and has a background thread that uses IndexReader.reopen() to keep a fresh version for reading?<br>
<br>
Also, what are the requirements? &nbsp;Must a document be visible to search within 10ms of being added? &nbsp;Or must it be visible to search from the time that the call to add it returns? &nbsp;In the latter case one might still use an approach like the above. &nbsp;Writing a small new segment to a RAMDirectory and then, with no merging, calling IndexReader.reopen(), should be quite fast. &nbsp;All merging could be done in the background, as should post-merge reopens() that involve large segments.<br>

<br>
In short, I wonder if new reader and writer implementations are in fact required or whether, perhaps with a few optimizations, the existing implementations might meet this need.<br>
<br>
Doug<br>
<br>
---------------------------------------------------------------------<div><div></div><div class="Wj3C7c"><br>
To unsubscribe, e-mail: <a href="mailto:java-dev-unsubscribe@lucene.apache.org" target="_blank">java-dev-unsubscribe@lucene.apache.org</a><br>
For additional commands, e-mail: <a href="mailto:java-dev-help@lucene.apache.org" target="_blank">java-dev-help@lucene.apache.org</a><br>
<br>
</div></div></blockquote></div><br>

------=_Part_117151_2806189.1230143028250--