lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Per Steffensen (JIRA)" <>
Subject [jira] [Updated] (SOLR-3178) Versioning - optimistic locking
Date Wed, 25 Jul 2012 12:22:37 GMT


Per Steffensen updated SOLR-3178:

    Attachment: SOLR-3173_3178_3382_3428_plus.patch

SOLR-3173_3178_3382_3428_plus.patch updated to fit on top of revision 1355667 on 4.x branch.

I have tried to describe for (almost) all files in the patch, which problem the changes in
them solve. Please see below. A lot of files has changed, but it is very small changes in
most of the files basically just making sure SolrRequestInfo-threadlocal is set in tests and
other stuff like that. The solutions to SOLR-3173 and SOLR-3178 are not very big. Actually
much more code has been done to solve SOLR-3382. With those descriptions I hope to convince
you to do the commit. Trust you test-suite - it has not been changed, except that additional
asserts has been added.

I am really not religious about whether the version-control code is in DistributedUpdateProcessor
or DirectUpdateHandler2, but I am religious about being backward compatible wrt such fundamental
changes in semantics, and I am religious about getting typed errors/exceptions back in responses,
so that I can react properly. I do believe version-control belong in DirectUpdateHandler2,
but if you want it in DistributedUpdateProcessor, basically just refactor (after commit) and
move the changes in DirectUpdateHandler2 to DistributedUpdateHandler.

* Solved SOLR-3173 and SOLR-3178 (implementation of fail-on-unique-key-violation and version-check),
by introducing 3 "modes" Solr server can run in (controlled by "semanticsMode" in solrconfig.xml
| updateHandler) - one for backward compatibility, one for a very strict version control,
and a hybrid (default).
** solr/core/src/java/org/apache/solr/update/ (the new features are
controlled from here, but basically using the rules in UpdateSemanticsMode. Got rid of duplicate
code in constructors - grrrr)
** solr/core/src/java/org/apache/solr/update/ (enum that encapsulates
the essence in the 3 possible "semanticsModes". Used by DirectUpdateHandler2. Nice "separation
of concern")
** solr/core/src/java/org/apache/solr/update/ (added leaderLogic calculated
(as isLeader) in DistributedUpdateProcesser to UpdateCommand so that it is available in DirectUpdateHandler2,
where it is used to know if the new checks must be performed or not. UpdateCommand also carrying
requestVersion, the version sent in the request)
** solr/core/src/java/org/apache/solr/core/ (control via configuration which
semantics-mode to use)
** solr/core/src/java/org/apache/solr/update/processor/ (maintains
leaderLogic on UpdateCommand)
** solr/core/src/java/org/apache/solr/update/ (maintains requestVersion
on UpdateCommand)
** solr/core/src/java/org/apache/solr/update/ (maintains requestVersion on UpdateCommand)
** solr/core/src/java/org/apache/solr/update/ (maintains leaderLogic and requestVersion
on UpdateCommand)
** solr/core/src/test-files/solr/collection1/conf/solrconfig-classic-semantics.xml (default
test solrconfig.xml but with "classic" semantics-mode, used to test that the new features
fail-on-unique-key-violation and version-check are not activated when running "classic" semantics
- the backward compatible "mode")
** solr/core/src/test-files/solr/collection1/conf/solrconfig-classic-consistency-hybrid-semantics.xml
(default test solrconfig.xml but with "classic-consistency-hybrid" semantics-mode. Actually
default solrconfig.xml could have been used since "classic-consistency-hybrid" semantics is
default, but used to test that it (also) works when this mode is selected explicitly)
** solr/test-framework/src/java/org/apache/solr/ and  solr/test-framework/src/java/org/apache/solr/util/
and solr/test-framework/src/java/org/apache/solr/util/ (basically just a lot
of changes making testing easier)

* Removed the (temporary and poor IMHO) implementation of fail-on-unique-key-violation and
version-check, but kept the introduced tests, so you can see that the same functionality is
still provided (on default semantics "classic-consistency-hybrid")
** solr/core/src/java/org/apache/solr/update/processor/
** solr/core/src/test/org/apache/solr/search/ (now we do not only get
exceptions in "bad" situations, we get typed exceptions telling us what is wrong)

* Solved SOLR-3382 - partRef and PartialError(s) introduced in multi-document-update-requests
so that they can be paired up with errors in responses, so that the client will know that
document-updates failed with what errors and therefore also which documents where updated
** solr/solrj/src/java/org/apache/solr/common/ (new generic interface to be
implemented by all parts of a request that is "just a part", and where an error that occures
during the handling of this particular part does not mean that handling other parts will also
result in error. Just to make it a little generic from the start, but by now only SolrInputDocuments
in update-requests are such "request-parts")
** solr/solrj/src/java/org/apache/solr/common/ (new default impl of RequestPart,
to be used by classes that really want to be a "request-part")
** solr/solrj/src/java/org/apache/solr/common/ (a "request part" implemented
using RequestPartImpl)
** solr/solrj/src/java/org/apache/solr/client/solrj/impl/ (dealing with
errors returned in responses. removed a poor initial but never finished attempt to transport
errors in responses, and replaced with my way of doing it (complete and finished solution).
this indicates a problem with the approach of only accepting small patches - you very easily
end up having partly implemented features that no-one ever finishes)
** solr/solrj/src/java/org/apache/solr/client/solrj/impl/ (dealing
with errors returned in responses)
** solr/solrj/src/java/org/apache/solr/common/partialerrors/ (new error type
indicating that the client is using the system in a wrong way - basically that a request is
inconsistent with server rules about how the request must be)
** solr/solrj/src/java/org/apache/solr/common/partialerrors/ (super-class
of all partial-error-types)
** solr/solrj/src/java/org/apache/solr/common/partialerrors/update/
(super-class of all partial-error-types that has to do with updates)
** solr/solrj/src/java/org/apache/solr/common/partialerrors/update/
(new error type indicating that the document you tried to create (not update) already exists)
** solr/solrj/src/java/org/apache/solr/common/partialerrors/update/
(new error type indicating that the document you tried to update (not create) does not already
exist (maybe it has been deleted))
** solr/solrj/src/java/org/apache/solr/common/partialerrors/update/ (new
error type indicating that there is a version-mismatch in the document that the client tries
to update. Basically the client loaded a version of the document, made some updates to it
and sent it to the server for storage of the changes, but the document on the server has changed
since the document was loaded by the client, and the client has therefore created his updated
document based on a obsolete version of the document. Nothing to do but reload, change and
attempt update again)
** solr/solrj/src/java/org/apache/solr/common/partialerrors/ (new error
type able to carry a map between "request parts" and the errors that occurred when they where
** solr/core/src/java/org/apache/solr/update/processor/ (catching
PartialErrors (defined as errors only related to the document currently being handled) and
continues with next document after adding it to response linked to document that triggered
the error)
** solr/core/src/java/org/apache/solr/servlet/ (encodes errors. If
multiple partial-errors occurred (e.g. multi-document-updates) a body is still needed in the
response, telling about the map between errors and the documents they occurred for. If only
single error occurred still just report that back using HTTP-error-code and reason-phrase.
Also set HTTP header including the error-type (namespace + name = java-package + java-class-name).
The actual encoding/decoding of errors in responses is isolated in SolrException)
** solr/solrj/src/java/org/apache/solr/common/ (encapsulating the logic
to encode/decode errors in responses - also visible for SolrJ clients so we can decode there
in a nice way)
** solr/core/src/java/org/apache/solr/response/ (able to carry information
about partial-errors and handled "parts" of a request)
** solr/core/src/java/org/apache/solr/handler/ and solr/core/src/java/org/apache/solr/handler/loader/
(a few constants - strings used to identify partRefs in requests - why two classes!?!)
** solr/core/src/java/org/apache/solr/handler/loader/ (partRef can be sent
in JSON requests)
** solr/core/src/java/org/apache/solr/handler/loader/ (partRefs can be sent
in CSV requests)
** solr/core/src/java/org/apache/solr/handler/loader/ (partRefs can be sent
in XML requests)
** solr/solrj/src/java/org/apache/solr/client/solrj/util/ (partRefs can be
sent in XML requests)
** solr/solrj/src/java/org/apache/solr/common/util/ (partRefs can be sent
in binary requests)
** solr/solrj/src/java/org/apache/solr/client/solrj/ (gives access to partial-errors,
map between "request parts" (essentially documents) and corresponding partial-errors, number
of partial-errors etc.)
** solr/solrj/src/java/org/apache/solr/client/solrj/response/ (see
** solr/test-framework/src/java/org/apache/solr/ and solr/test-framework/src/java/org/apache/solr/util/
and solr/test-framework/src/java/org/apache/solr/util/ (return errors in helper-assertFailed-methods,
so that those errors can be inspected in further more specific assertions - basically just
a lot of changes making testing easier)
** solr/solrj/src/test/org/apache/solr/client/solrj/ (nicer asserts including
asserting on correct type (class) of exception)
** solr/core/src/test/org/apache/solr/handler/ (test of it when using JSON
requests and responses)
** solr/core/src/test/org/apache/solr/handler/

* Solved SOLR-3428
** solr/core/src/java/org/apache/solr/update/

* In DirectUpdateHandler2 (solving SOLR-3173 and SOLR-3178) and other places Im working with
the SolrQueryRequest and SolrQueryResponse found in the threadlocal of SolrRequestInfo, so
therefore this threadlocal needs to be set (and cleared) in many tests
** solr/core/src/test/org/apache/solr/update/processor/
** solr/core/src/test/org/apache/solr/update/processor/
** solr/core/src/test/org/apache/solr/update/processor/
** solr/core/src/test/org/apache/solr/update/
** solr/core/src/test/org/apache/solr/request/
** solr/core/src/test/org/apache/solr/request/
** solr/core/src/test/org/apache/solr/core/
** solr/core/src/test/org/apache/solr/
** solr/core/src/test/org/apache/solr/
** solr/core/src/test/org/apache/solr/
** solr/core/src/test/org/apache/solr/search/function/
** solr/core/src/test/org/apache/solr/search/
** solr/core/src/test/org/apache/solr/search/
** solr/core/src/test/org/apache/solr/search/
** solr/core/src/test/org/apache/solr/search/
** solr/core/src/test/org/apache/solr/cloud/
** solr/core/src/test/org/apache/solr/highlight/
** solr/core/src/test/org/apache/solr/highlight/
** solr/core/src/test/org/apache/solr/highlight/ 
** solr/core/src/test/org/apache/solr/
** solr/core/src/test/org/apache/solr/
** solr/core/src/test/org/apache/solr/handler/
** solr/core/src/test/org/apache/solr/handler/
**  solr/core/src/test/org/apache/solr/handler/component/
** solr/core/src/test/org/apache/solr/handler/
** solr/core/src/java/org/apache/solr/servlet/
** solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/
** solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/
** solr/contrib/uima/src/test/org/apache/solr/uima/processor/
** solr/test-framework/src/java/org/apache/solr/
** solr/test-framework/src/java/org/apache/solr/util/
** solr/test-framework/src/java/org/apache/solr/util/

* SolrRequest sub-classes has a lot of duplicated code (grrrr). Cleaned up and put shared
code in one place - SolrRequest
** solr/solrj/src/java/org/apache/solr/client/solrj/ 
** solr/solrj/src/java/org/apache/solr/client/solrj/request/
** solr/solrj/src/java/org/apache/solr/client/solrj/request/
** solr/solrj/src/java/org/apache/solr/client/solrj/request/
** solr/solrj/src/java/org/apache/solr/client/solrj/request/
** solr/solrj/src/java/org/apache/solr/client/solrj/request/
** solr/solrj/src/java/org/apache/solr/client/solrj/request/
** solr/solrj/src/java/org/apache/solr/client/solrj/request/
** solr/solrj/src/java/org/apache/solr/client/solrj/request/

* Think we should work with a common (for both server and client side) "_version_" String
constant, but VersionInfo is not available for client side, so I removed VERSION_FIELD = "_version_"
from VersionInfo to SolrInputDocument in order to make it available for both server and client
(SolrJ) side.
** solr/core/src/java/org/apache/solr/update/
** solr/solrj/src/java/org/apache/solr/common/
** solr/core/src/java/org/apache/solr/update/
** solr/core/src/java/org/apache/solr/update/processor/
** solr/core/src/test/org/apache/solr/update/
** solr/core/src/test/org/apache/solr/cloud/

* Added some new tests with focus on testing the new version- and unique-key-based features
- ability to fail instead of overwriting on unique-key violation and ability to report errors
on version conflict
** solr/core/src/test/org/apache/solr/update/
** solr/core/src/test/org/apache/solr/update/
** solr/core/src/test/org/apache/solr/update/
** solr/core/src/test/org/apache/solr/cloud/
** solr/solrj/src/test/org/apache/solr/client/update/
** solr/solrj/src/test/org/apache/solr/client/update/

* Renamed AddUpdateCommand.overwrite to AddUpdateCommand.classicOverwrite, because this is
a field that is now only used when running with "classic" semantics
** solr/core/src/java/org/apache/solr/update/
** solr/core/src/java/org/apache/solr/update/
**  solr/core/src/java/org/apache/solr/handler/loader/
** solr/core/src/java/org/apache/solr/handler/loader/
** solr/core/src/java/org/apache/solr/handler/loader/
** solr/core/src/java/org/apache/solr/handler/loader/
** solr/contrib/extraction/src/java/org/apache/solr/handler/extraction/
** solr/solrj/src/java/org/apache/solr/client/solrj/request/
**  solr/core/src/java/org/apache/solr/update/ 
** solr/core/src/test/org/apache/solr/update/
** solr/core/src/test/org/apache/solr/handler/
** solr/core/src/test/org/apache/solr/handler/
** etc

* When putting config-files into ZK you can have them named something different in there than
their file-name on disk. E.g. ./my/folder/my-solrconfig.xml can be named solrconfig.xml in
** solr/core/src/test/org/apache/solr/cloud/
** solr/core/src/test/org/apache/solr/cloud/

* Misc unimportant cleanup - removed unneeded imports, etc.
** solr/core/src/test/org/apache/solr/update/
** solr/core/src/test/org/apache/solr/request/ (just a nicer way to look
and afterwards use rsp.getException())
** solr/core/src/java/org/apache/solr/servlet/ (just a nicer way
to look and afterwards use rsp.getException())
** solr/core/src/java/org/apache/solr/client/solrj/embedded/ (just
a nicer way to look and afterwards use rsp.getException())
**  solr/solrj/src/java/org/apache/solr/client/solrj/impl/

> Versioning - optimistic locking
> -------------------------------
>                 Key: SOLR-3178
>                 URL:
>             Project: Solr
>          Issue Type: New Feature
>          Components: update
>    Affects Versions: 3.5
>         Environment: All
>            Reporter: Per Steffensen
>            Assignee: Per Steffensen
>              Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey, update, versioning
>             Fix For: 4.0
>         Attachments: SOLR-3173_3178_3382_3428_plus.patch, SOLR-3173_3178_3382_3428_plus.patch,
SOLR-3178.patch, SOLR_3173_3178_3382_plus.patch
>   Original Estimate: 168h
>  Remaining Estimate: 168h
> In order increase the ability of Solr to be used as a NoSql database (lots of concurrent
inserts, updates, deletes and queries in the entire lifetime of the index) instead of just
a search index (first: everything indexed (in one thread), after: only queries), I would like
Solr to support versioning to be used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need to provide
a version-number equal to "the version number I got when I fetched the existing document for
update" plus one. If this provided version-number does not correspond to "the newest version-number
of that document at the time of update" plus one, I will get a VersionConflict error. If it
does correspond the document will be updated with the new one, so that "the newest version-number
of that document" is NOW one higher than before the update. Correct but efficient concurrency
> When my intent (see SOLR-3173) is to insert a new document, the version number provided
will not be used - instead a version-number 0 will be used. According to SOLR-3173 insert
will only succeed if a document with the same value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of course we
need to be able to identify when a document "is the same" - that, per definition, is when
the values of the uniqueKey-fields are equal. 
> The functionality provided by this issue is only really meaningfull when you run with
"updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and only one single
SVN patch might be given to cover both issues.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message