lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hendrik Haddorp <hendrik.hadd...@gmx.net>
Subject Re: atomic updates in conjunction with optimistic concurrency
Date Fri, 21 Jul 2017 22:08:09 GMT
Thanks for trying to reproduce my issue.

I'm using a Solr Cloud, my collection was quite small, only a 50-500 
documents, with one shard and a replication factor of 3. I updated all 
of the documents in one request. Beside that the flow is pretty much 
like yours.

The goal of my code was to add a field to all documents that did not 
contain the field already, which should actually have been every 
document in the collection. So the code does a query and gets a few 
fields for each document. Then I use an atomic update to add two fields. 
In the end I send all fields to Solr in one request. The whole thing did 
then loop over a few hundred collections. In most cases it worked just 
fine but in some it failed. If so I could reproduce the issue on that 
collection constantly. In an earlier version I accidentally tried to add 
a field instead of setting it and that had caused an exception as the 
field only allowed a single value. At least in one case the version 
conflict later on happened on a collection that had this issue before. 
don't really think that causes it so just stating for the sake of 
completeness. I was not using a document centric version field but the 
normal _version_ field.

Even though I'm running on Solr 6.3 my config still contains 
"<luceneMatchVersion>6.2.1</luceneMatchVersion>". Not sure if that might 
have an effect.

The main code is this:
                                     SolrQuery query = new SolrQuery();
query.setQuery(documentsMatchingQueryQ);
query.setRows(Integer.MAX_VALUE);
query.setFields(documentsMatchingQueryFL);

                                     QueryResponse queryResponse = 
solr.query(collectionName, query);
                                     if (queryResponse.getStatus() != 0) {
                                         LOGGER.error("request failed, 
status: ", queryResponse);
                                         throw new 
IllegalStateException("request failed");
                                     }
                                     LOGGER.info("{} | documents to 
process: {}", collectionName, queryResponse.getResults().getNumFound());
                                     if 
(queryResponse.getResults().getNumFound() > 0) {
                                         LOGGER.trace("{} | start 
processing", collectionName);
List<SolrInputDocument> docs = queryResponse.getResults().stream()
.map(documentGenerator::apply)
                                                 .peek(doc -> 
LOGGER.trace("{} doc: {}", collectionName, doc))
.collect(Collectors.toList());
                                         if (!reportOnly) {
                                             LOGGER.trace("{} | updating 
solr", collectionName);
                                             solr.add(collectionName, docs);
solr.commit(collectionName, false, false, true);

                                         }
                                         LOGGER.trace("{} | processing 
done", collectionName);
                                     }

The documentGenerator looks like this:
originalDoc -> {
                     SolrInputDocument doc = new SolrInputDocument();

                     doc.addField("id", originalDoc.getFieldValue("id"));
                     doc.addField("__docId__", 
Collections.singletonMap("set", 
originalDoc.getFieldValue("classification") + ":" + 
originalDoc.getFieldValue("id")));
                     doc.addField("systemModified", 
Collections.singletonMap("set", originalDoc.getFieldValue("lastModified")));
                     doc.addField("_version_", 
originalDoc.getFieldValue("_version_"));

                     return doc;
                 }

On 21.07.2017 22:33, Amrit Sarkar wrote:
> Hendrik,
>
> Ran a little test on 6.3, with infinite atomic updates with optimistic
> concurrency,
> cannot *reproduce*:
>
> List<SolrInputDocument> docs = new ArrayList<>();
>> SolrInputDocument document = new SolrInputDocument();
>> document.addField("id", String.valueOf(1));
>> document.addField("external_version_field_s", System.currentTimeMillis()); // normal
update
>> docs.add(document);
>> UpdateRequest updateRequest = new UpdateRequest();
>> updateRequest.add(docs);
>> client.request(updateRequest, collection);
>> updateRequest = new UpdateRequest();
>> updateRequest.commit(client, collection);
>>
>> while (true) {
>>      QueryResponse response = client.query(new ModifiableSolrParams().add("q", "id:1"));
>>      System.out.println(response.getResults().get(0).get("_version_"));
>>      docs = new ArrayList<>();
>>      document = new SolrInputDocument();
>>      document.addField("id", String.valueOf(1));
>>      Map<String, String> map = new HashMap<>();
>>      map.put("set", createSentance(1)); // atomic map value
>>      document.addField("external_version_field_s", map);
>>      document.addField("_version_", response.getResults().get(0).get("_version_"));
>>      docs.add(document);
>>      updateRequest = new UpdateRequest();
>>      updateRequest.add(docs);
>>      client.request(updateRequest, collection);
>>      updateRequest = new UpdateRequest();
>>      updateRequest.commit(client, collection);
>> }
>>
>> Maybe you can let us know more details how the update been made?
> Amrit Sarkar
> Search Engineer
> Lucidworks, Inc.
> 415-589-9269
> www.lucidworks.com
> Twitter http://twitter.com/lucidworks
> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
>
> On Fri, Jul 21, 2017 at 10:36 PM, Hendrik Haddorp <hendrik.haddorp@gmx.net>
> wrote:
>
>> Hi,
>>
>> I can't find anything about this in the Solr logs. On the caller side I
>> have this:
>> Error from server at http://xxxxx_shard1_replica2: version conflict for
>> xxxxx expected=1573538179623944192 actual=1573546159565176832
>> org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error
>> from server at http://xxxxx_shard1_replica2: version conflict for xxxxx
>> expected=1573538179623944192 actual=1573546159565176832
>>      at org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:765)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1173)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at org.apache.solr.client.solrj.impl.CloudSolrClient.requestWit
>> hRetryOnStaleState(CloudSolrClient.java:1062)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1004)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:106)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at org.apache.solr.client.solrj.SolrClient.add(SolrClient.java:71)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      ...
>> Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
>> Error from server at http://xxxxx_shard1_replica2: version conflict for
>> xxxxx expected=1573538179623944192 actual=1573546159565176832
>>      at org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:593)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:435)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:387)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at org.apache.solr.client.solrj.impl.CloudSolrClient.lambda$dir
>> ectUpdate$0(CloudSolrClient.java:742) ~[solr-solrj-6.3.0.jar:6.3.0
>> a66a44513ee8191e25b477372094bfa846450316 - shalin - 2016-11-02 19:52:43]
>>      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[?:1.8.0_131]
>>      at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolE
>> xecutor.lambda$execute$0(ExecutorUtil.java:229)
>> ~[solr-solrj-6.3.0.jar:6.3.0 a66a44513ee8191e25b477372094bfa846450316 -
>> shalin - 2016-11-02 19:52:43]
>>      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[?:1.8.0_131]
>>      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> ~[?:1.8.0_131]
>>      at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_131]
>>
>> The version "1573546159565176832" does not exist. It looks a bit like the
>> update was first creating a new value and then checks against it.
>>
>> regards,
>> Hendrik
>>
>>
>> On 21.07.2017 18:21, Amrit Sarkar wrote:
>>
>>> Hendrik,
>>>
>>> Can you list down the error snippet so that we can refer the code where
>>> exactly that is happening.
>>>
>>>
>>> Amrit Sarkar
>>> Search Engineer
>>> Lucidworks, Inc.
>>> 415-589-9269
>>> www.lucidworks.com
>>> Twitter http://twitter.com/lucidworks
>>> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
>>>
>>> On Fri, Jul 21, 2017 at 9:50 PM, Hendrik Haddorp <hendrik.haddorp@gmx.net
>>> wrote:
>>>
>>> Hi,
>>>> when I try to use an atomic update in conjunction with optimistic
>>>> concurrency Solr sometimes complains that the version I passed in does
>>>> not
>>>> match. The version in my request however match to what is stored and what
>>>> the exception states as the actual version does not exist in the
>>>> collection
>>>> at all. Strangely this does only happen sometimes but once it happens
>>>> for a
>>>> collection it seems to stay like that. Any idea why that might happen?
>>>>
>>>> I'm using Solr 6.3 in Cloud mode with SolrJ.
>>>>
>>>> regards,
>>>> Hendrik
>>>>
>>>>


Mime
View raw message