Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of
 simon.willnauer@googlemail.com designates 209.85.161.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=googlemail.com; s=gamma;
        h=mime-version:reply-to:in-reply-to:references:date:message-id
         :subject:from:to:content-type:content-transfer-encoding;
        b=lNCbWUnvwImAv19S5trUdRYBeqSnqlMwK1X34utquVE8trCLVG2eVL2v0enoyzKzC5
         3dumbphmvivwoIFKnL/jXllEAMI1yOagtCvEA6/Zvg3vObraw8wc+b01NAcblw3dF5m2
         588qHGL8doC3Yp4XBDMWPVsi0LYFq+bzCsGfA=
MIME-Version: 1.0
Reply-To: simon.willnauer@gmail.com
In-Reply-To: 
 <CF3CE3EFBCA3564185DF065952A267C8534369A85D@NOK-EUMSG-01.mgdnok.nokia.com>
References: 
 <CF3CE3EFBCA3564185DF065952A267C8534369A5EA@NOK-EUMSG-01.mgdnok.nokia.com>
	 <4BFAD489.6050108@gmail.com>
	 <CF3CE3EFBCA3564185DF065952A267C8534369A85D@NOK-EUMSG-01.mgdnok.nokia.com>
Date: Mon, 24 May 2010 22:29:04 +0200
Message-ID: <AANLkTinrkFpap-GZJCELUTCQT8u_jH6ObaVjGQQrlwKX@mail.gmail.com>
Subject: Re: Solr updateRequestHandler and performance vs. atomicity
From: Simon Willnauer <simon.willnauer@googlemail.com>
To: dev@lucene.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Karl,

what are you describing seems to be a good usecase for something like
a message queue where you push a document or record to a queue which
guarantees the queues persistence. I look at this from a little
different perspective, in a distributed environment you would have to
guarantee delivery to a single solr instance but on several or at
least n instances but that is a different story.

>From a Solr point of view this sounds like a need for a write-ahead
log that guarantees durability and atomicity. I like this idea as it
might also solve lots of problems in distributed environments (solr
cloud) etc.

Very interesting topic - should investigate more in this direction....


simon


On Mon, May 24, 2010 at 10:03 PM,  <karl.wright@nokia.com> wrote:
> Hi Mark,
>
> Unfortunately, indexing performance *is* of concern, otherwise I'd alread=
y be committing on every post.
>
> If your guess is correct, you are basically saying that adding a document=
 to an index in Solr/Lucene is just as fast as writing that file directly t=
o the disk. =C2=A0Because, obviously, if we want guaranteed delivery, that'=
s what we'd have to do. =C2=A0But I think this is worth the experiment - So=
lr/Lucene may be fast, but I have doubts that it can perform as well as raw=
 disk I/O and still manage to do anything in the way of document analysis o=
r (heaven forbid) text extraction.
>
>
>
> -----Original Message-----
> From: ext Mark Miller [mailto:markrmiller@gmail.com]
> Sent: Monday, May 24, 2010 3:33 PM
> To: dev@lucene.apache.org
> Subject: Re: Solr updateRequestHandler and performance vs. atomicity
>
> On 5/24/10 3:10 PM, karl.wright@nokia.com wrote:
>> Hi all,
>> It seems to me that the "commit" logic in the Solr updateRequestHandler
>> (or wherever the logic is actually located) conflates two different
>> semantics. One semantic is what you need to do to make the index process
>> perform well. The other semantic is guaranteed atomicity of document
>> reception by Solr.
>> In particular, it would be nice to be able to post documents in such a
>> way that you can guarantee that the document is permanently in Solr's
>> queue, safe in the event of a Solr restart, etc., even if the document
>> has not yet been "committed".
>> This issue came up in the LCF talk that I gave, and I initially thought
>> that separating the two kinds of events would necessarily be an LCF
>> change, but the more I thought about it the more I realized that other
>> Solr indexing clients may also benefit from such a separation.
>> Does anyone agree? Where should this logic properly live?
>> Thanks,
>> Karl
>
> Its an interesting idea - but I think you would likely pay a similar
> cost to guarantee reception as you would to commit (also, I'm not sure
> Lucene guarantees it - it works for consistency, but I'm not so sure it
> achieves durability).
>
> I can think of two things offhand -
>
> Perhaps store the text and use fsync to quasi guarantee acceptance -
> then index from the store on the commit.
>
> Another simpler idea if only the separation is important and not the
> performance - index to another side index, taking advantage of Lucene's
> current commit functionality, and then use addIndex to merge to the main
> index on commit.
>
> Just spit balling though.
>
> I think this would obviously need to be an optional mode.
>
> --
> - Mark
>
> http://www.lucidimagination.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org