lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hendrik Haddorp <hendrik.hadd...@gmx.net>
Subject Re: Soft commit and reading data just after the commit
Date Mon, 19 Dec 2016 19:31:06 GMT
Hi,

the SolrJ API has this method: SolrClient.commit(String collection, 
boolean waitFlush, boolean waitSearcher, boolean softCommit).
My assumption so far was that when you set waitSearcher to true that the 
method call only returns once a search would find the new data, which 
sounds what you want. I used this already and it seemed to work just fine.

regards,
Hendrik

On 19.12.2016 04:09, Lasitha Wattaladeniya wrote:
> Hi all,
>
> Thanks for your replies,
>
> @dorian : the requirement is,  we are showing a list of entries on a page.
> For each user there's a read / unread flag.  The data for listing is
> fetched from solr. And you can see the entry was previously read or not. So
> when a user views an entry by clicking.  We are updating the database flag
> to READ and use real time indexing to update solr entry.  So when the user
> close the full view of the entry and go back to entry listing page,  the
> data fetched from solr should be updated to READ. That's the use case we
> are trying to fix.
>
> @eric : thanks for the lengthy reply.  So let's say I increase the
> autosoftcommit time out to may be 100 ms.  In that case do I have to wait
> much that time from client side before calling search ?.  What's the
> correct way of achieving this?
>
> Regards,
> Lasitha
>
> On 18 Dec 2016 23:52, "Erick Erickson" <erickerickson@gmail.com> wrote:
>
>> 1 ms autocommit is far too frequent. And it's not
>> helping you anyway.
>>
>> There is some lag between when a commit happens
>> and when the docs are really available. The sequence is:
>> 1> commit (soft or hard-with-opensearcher=true doesn't matter).
>> 2> a new searcher is opened and autowarming starts
>> 3> until the new searcher is opened, queries continue to be served by
>> the old searcher
>> 4> the new searcher is fully opened
>> 5> _new_ requests are served by the new searcher.
>> 6> the last request is finished by the old searcher and it's closed.
>>
>> So what's probably happening is that you send docs and then send a
>> query and Solr is still in step <3>. You can look at your admin UI
>> pluginst/stats page or your log to see how long it takes for a
>> searcher to open and adjust your expectations accordingly.
>>
>> If you want to fetch only the document (not try to get it by a
>> search), Real Time Get is designed to insure that you always get the
>> most recent copy whether it's searchable or not.
>>
>> All that said, Solr wasn't designed for autocommits that are that
>> frequent. That's why the documentation talks about _Near_ Real Time.
>> You may need to adjust your expectations.
>>
>> Best,
>> Erick
>>
>> On Sun, Dec 18, 2016 at 6:49 AM, Dorian Hoxha <dorian.hoxha@gmail.com>
>> wrote:
>>> There's a very high probability that you're using the wrong tool for the
>>> job if you need 1ms softCommit time. Especially when you always need it
>> (ex
>>> there are apps where you need commit-after-insert very rarely).
>>>
>>> So explain what you're using it for ?
>>>
>>> On Sun, Dec 18, 2016 at 3:38 PM, Lasitha Wattaladeniya <
>> wattale@gmail.com>
>>> wrote:
>>>
>>>> Hi Furkan,
>>>>
>>>> Thanks for the links. I had read the first one but not the second one. I
>>>> did read it after you sent. So in my current solrconfig.xml settings
>> below
>>>> are the configurations,
>>>>
>>>> <autoSoftCommit>
>>>>     <maxTime>${solr.autoSoftCommit.maxTime:1}</maxTime>
>>>>   </autoSoftCommit>
>>>>
>>>>
>>>> <autoCommit>
>>>>     <maxTime>15000</maxTime>
>>>>     <openSearcher>false</openSearcher>
>>>>   </autoCommit>
>>>>
>>>> The problem i'm facing is, just after adding the documents to solr using
>>>> solrj, when I retrieve data from solr I am not getting the updated
>> results.
>>>> This happens time to time. Most of the time I get the correct data but
>> in
>>>> some occasions I get wrong results. so as you suggest, what the best
>>>> practice to use here ? , should I wait 1 mili second before calling for
>>>> updated results ?
>>>>
>>>> Regards,
>>>> Lasitha
>>>>
>>>> Lasitha Wattaladeniya
>>>> Software Engineer
>>>>
>>>> Mobile : +6593896893
>>>> Blog : techreadme.blogspot.com
>>>>
>>>> On Sun, Dec 18, 2016 at 8:46 PM, Furkan KAMACI <furkankamaci@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Lasitha,
>>>>>
>>>>> First of all, did you check these:
>>>>>
>>>>> https://cwiki.apache.org/confluence/display/solr/Near+
>>>> Real+Time+Searching
>>>>> https://lucidworks.com/blog/2013/08/23/understanding-
>>>>> transaction-logs-softcommit-and-commit-in-sorlcloud/
>>>>>
>>>>> after that, if you cannot adjust your configuration you can give more
>>>>> information and we can find a solution.
>>>>>
>>>>> Kind Regards,
>>>>> Furkan KAMACI
>>>>>
>>>>> On Sun, Dec 18, 2016 at 2:28 PM, Lasitha Wattaladeniya <
>>>> wattale@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi furkan,
>>>>>>
>>>>>> Thanks for your reply, it is generally a query heavy system. We are
>>>> using
>>>>>> realtime indexing for editing the available data
>>>>>>
>>>>>> Regards,
>>>>>> Lasitha
>>>>>>
>>>>>> Lasitha Wattaladeniya
>>>>>> Software Engineer
>>>>>>
>>>>>> Mobile : +6593896893 <+65%209389%206893>
>>>>>> Blog : techreadme.blogspot.com
>>>>>>
>>>>>> On Sun, Dec 18, 2016 at 8:12 PM, Furkan KAMACI <
>> furkankamaci@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Lasitha,
>>>>>>>
>>>>>>> What is your indexing / querying requirements. Do you have an
index
>>>>>>> heavy/light  - query heavy/light system?
>>>>>>>
>>>>>>> Kind Regards,
>>>>>>> Furkan KAMACI
>>>>>>>
>>>>>>> On Sun, Dec 18, 2016 at 11:35 AM, Lasitha Wattaladeniya <
>>>>>>> wattale@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hello devs,
>>>>>>>>
>>>>>>>> I'm here with another problem i'm facing. I'm trying to do
a
>> commit
>>>>>>> (soft
>>>>>>>> commit) through solrj and just after the commit, retrieve
the data
>>>> from
>>>>>>>> solr (requirement is to get updated data list).
>>>>>>>>
>>>>>>>> I'm using soft commit instead of the hard commit, is previously
I
>> got
>>>>>>> an
>>>>>>>> error "Exceeded limit of maxWarmingSearchers=2, try again
later"
>>>>>>> because of
>>>>>>>> too many commit requests. Now I have removed the explicit
commit
>> and
>>>>>>> has
>>>>>>>> let the solr to do the commit using autoSoftCommit *(1 mili
>> second)*
>>>>>>> and
>>>>>>>> autoCommit *(30 seconds)* configurations. Now I'm not getting
any
>>>>>>> errors
>>>>>>>> when i'm committing frequently.
>>>>>>>>
>>>>>>>> The problem i'm facing now is, I'm not getting the updated
data
>> when
>>>> I
>>>>>>>> fetch from solr just after the soft commit. So in this case
what
>> are
>>>>>>> the
>>>>>>>> best practices to use ? to wait 1 mili second before retrieving
>> data
>>>>>>> after
>>>>>>>> soft commit ? I don't feel like waiting from client side
is a good
>>>>>>> option.
>>>>>>>> Please give me some help from your expert knowledge
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>> Lasitha Wattaladeniya
>>>>>>>> Software Engineer
>>>>>>>>
>>>>>>>> Mobile : +6593896893
>>>>>>>> Blog : techreadme.blogspot.com
>>>>>>>>
>>>>>>


Mime
View raw message