lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Per Steffensen (JIRA)" <>
Subject [jira] [Commented] (SOLR-3178) Versioning - optimistic locking
Date Thu, 07 Jun 2012 07:40:25 GMT


Per Steffensen commented on SOLR-3178:

{quote} It's just a guess, but I think it unlikely any committers would feel comfortable tackling
this big patch, or even have time to understand all of the different aspects. They may agree
with some parts but disagree with other parts. Small, tightly focused patches that tackle
a single aspect are much more likely to get attention.{quote}

I understand, but things are very related, so its hard for me to go back and divide into smaller
patches and feed them one by one to you. Things are related because "optimistic locking"-errors
and "unique key constraints"-errors are not worth much if exact information about what went
wrong does not go back to the client (therefore adding the feature of sending typed errors
back to the client is needed). And the two kinds on new errros are the first "single document"-related
errors in Solr, and therefore it, for the first time in history of Solr, is very nice to have
the ability to keep processing documents in a multi-document-update-request even though one
document-update fails with one of those errors (therefore multi-error responses with references
on each error to the document it relates to is needed). And backward compability is just always
nice, and it this case required, IMHO

{quote} Just speaking for myself, when I get time to look at optimistic locking stuff again,
I'll probably browse your latest patch to see what improvements I can cherrypick (assuming
no one else gets around to making some smaller patches first). {quote}

I would really like that, and I will be available for any help you need. Or just make me a
committer, and you will live happily ever after :-) I am very experienced doing this kind
of stuff.

BTW, I saw that you committed some "inc" stuff on this JIRA. As I understand it, it is a new
feature where you are able to tell server-side Solr to add to a field, instead of doing it
yourself on client side. Server-side Solr can then do it while having the lock on the document,
and therefore no optimistic locking is needed in principle - but only if your updates use
"inc"s only. I think it deserves its own JIRA, because it doesnt have anything to do with
optimistic locking, except that you do not need optimistic locking if your updates use "inc"s
only. It is probably not realistic that any real world application would ever be able to do
its updates based on "inc"s alone, so in isolation I think the feature is kinda useless. But
if you generalize on the feature it could be great. Generalize in a way where you can have
any update-semantic performed on your document server-side (while having the document lock),
then utilizing this feature heavily would enable you to avoid using optimistic locking and
that might be prefered in some situations - just even another reason while backward compability
on optimistic locking is nice, so that you can have it disabled (even though you would like
to have a full-featured Cloud-setup with updateLog, _version_ field etc.) if you are not going
to use it. 

I imagine that you can install document(-field)-manipulating primitives on server-side Solr
and associate them with a names (e.g. associate the name "inc" with a piece of code adding
to a field, "multiply" with a piece of code multiplying to a field, "string-concat" with a
piece of code that concats to fields into a third field, etc) and then have pairs of document-field
and document(-field)-manipulating-ref in your updates (e.g. send updates saying "update document
A manipulate fields inc(field1,13), multiply(field2,4), string-concat(field3,field4,field5)").
But as long as "inc"s is the only such manipulate-on-server-side-while-having-the-lock-primitive
it is kinda useless. The code you associate with primitives ("inc", "multiply", "string-concat"
etc) could just be a small piece of Java code, and the association needs to go on in solrconfig.xml.
Or if you are really cool (and you trust your clients are not doing SQL-injection-ish hacks
on you) you can allow clients to install primitives on-demand, either as part of update-requests
that use those primitives or as separate primitive-registration-request that you do before
you start using a primitive. Such request-carried primitives could be in the form of Groovy
code, Ruby code, JavaScript code, or whatever the JVM is able to compile and use on-demand.

Regards, Per Steffensen

> Versioning - optimistic locking
> -------------------------------
>                 Key: SOLR-3178
>                 URL:
>             Project: Solr
>          Issue Type: New Feature
>          Components: update
>    Affects Versions: 3.5
>         Environment: All
>            Reporter: Per Steffensen
>            Assignee: Per Steffensen
>              Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
>             Fix For: 4.0
>         Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>   Original Estimate: 168h
>  Remaining Estimate: 168h
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent
inserts, updates, deletes and queries in the entire lifetime of the index) instead of just
a search index (first: everything indexed (in one thread), after: only queries), I would like
Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide
a version-number equal to "the version number I got when I fetched the existing document for
update" plus one. If this provided version-number does not correspond to "the newest version-number
of that document at the time of update" plus one, I will get a VersionConflict error. If it
does correspond the document will be updated with the new one, so that "the newest version-number
of that document" is NOW one higher than before the update. Correct but efficient concurrency
> When my intent (see SOLR-3173) is to insert a new document, the version number provided
will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert
will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we
need to be able to identify when a document "is the same" - that, per definition, is when
the values of the uniqueKey-fields are equal. 
> The functionality provided by this issue is only really meaningfull when you run with
"updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single
SVN patch might be given to cover both issues.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message