lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alessandro Benedetti <abenede...@apache.org>
Subject Re: commit it taking 1300 ms
Date Mon, 22 Aug 2016 14:11:42 GMT
I agree with the suggestions so far.
The cache auto-warming doesn't seem the problem as the index is not massive
and the auto-warm is for only 10 docs.
Are you using any warming query for the new searcher ?

Are you using soft or hard commit ?
This can make the difference ( soft are much cheaper, not free but cheaper)
.
You said :
" Actually earlier it was taking less but suddenly it has increased "

What happened ?
Anyway, there are a lot of questions to answer before we can help you...

Cheers

On Fri, Aug 12, 2016 at 4:58 AM, Esther-Melaine Quansah <
esther.quansah@lucidworks.com> wrote:

> Midas,
>
> I’d like further clarification as well. Are you sending commits along with
> each document that you’re POSTing to Solr? If so, you’re essentially either
> opening a new searcher or flushing to disk with each POST which could
> explain latency between each request.
>
> Thanks,
>
> Esther
> > On Aug 11, 2016, at 12:19 PM, Erick Erickson <erickerickson@gmail.com>
> wrote:
> >
> > bq:  we post json documents through the curl it takes the time (same
> time i
> > would like to say that we are not hard committing ). that curl takes time
> > i.e. 1.3 sec.
> >
> > OK, I'm really confused. _what_ is taking 1.3 seconds? When you said
> > commit, I was thinking of Solr's commit operation, which is totally
> distinct
> > from just adding a doc to the index. But I read the above statement
> > as you're saying it takes 1.3 seconds just to send a doc to Solr.
> >
> > Let's see the exact curl command you're using please?
> >
> > Best,
> > Erick
> >
> >
> > On Thu, Aug 11, 2016 at 5:32 AM, Emir Arnautovic
> > <emir.arnautovic@sematext.com> wrote:
> >> Hi Midas,
> >>
> >> 1. How many indexing threads?
> >> 2. Do you batch documents and what is your batch size?
> >> 3. How frequently do you commit?
> >>
> >> I would recommend:
> >> 1. Move commits to Solr (set auto soft commit to max allowed time)
> >> 2. Use batches (bulks)
> >> 3. tune bulk size and number of threads to achieve max performance.
> >>
> >> Thanks,
> >> Emir
> >>
> >>
> >>
> >> On 11.08.2016 08:21, Midas A wrote:
> >>>
> >>> Emir,
> >>>
> >>> other queries:
> >>>
> >>> a) Solr cloud : NO
> >>> b) <filterCache class="solr.FastLRUCache"
> >>> size="5000" initialSize="5000" autowarmCount="10"/>
> >>> c)  <queryResultCache class="solr.LRUCache"
> >>> size="1000" initialSize="1000" autowarmCount="10"/>
> >>> d) <documentCache class="solr.LRUCache"
> >>> size="1000" initialSize="1000" autowarmCount="10"/>
> >>> e) we are using multi threaded system.
> >>>
> >>> On Thu, Aug 11, 2016 at 11:48 AM, Midas A <test.midas@gmail.com>
> wrote:
> >>>
> >>>> Emir,
> >>>>
> >>>> we post json documents through the curl it takes the time (same time
i
> >>>> would like to say that we are not hard committing ). that curl takes
> time
> >>>> i.e. 1.3 sec.
> >>>>
> >>>> On Wed, Aug 10, 2016 at 2:29 PM, Emir Arnautovic <
> >>>> emir.arnautovic@sematext.com> wrote:
> >>>>
> >>>>> Hi Midas,
> >>>>>
> >>>>> According to your autocommit configuration and your worry about
> commit
> >>>>> time I assume that you are doing explicit commits from client code
> and
> >>>>> that
> >>>>> 1.3s is client observed commit time. If that is the case, than it
> might
> >>>>> be
> >>>>> opening searcher that is taking time.
> >>>>>
> >>>>> How do you index data - single threaded or multithreaded? How
> frequently
> >>>>> do you commit from client? Can you let Solr do soft commits instead
> of
> >>>>> explicitly committing? Do you have warmup queries? Is this SolrCloud?
> >>>>> What
> >>>>> is number of servers (what spec), shards, docs?
> >>>>>
> >>>>> In any case monitoring can give you more info about server/Solr
> behavior
> >>>>> and help you diagnose issues more easily/precisely. One such
> monitoring
> >>>>> tool is our SPM <http://sematext.com/spm>.
> >>>>>
> >>>>> Regards,
> >>>>> Emir
> >>>>>
> >>>>> --
> >>>>> Monitoring * Alerting * Anomaly Detection * Centralized Log
> Management
> >>>>> Solr & Elasticsearch Support * http://sematext.com/
> >>>>>
> >>>>> On 10.08.2016 05:20, Midas A wrote:
> >>>>>
> >>>>>> Thanks for replying
> >>>>>>
> >>>>>> index size:9GB
> >>>>>> 2000 docs/sec.
> >>>>>>
> >>>>>> Actually earlier it was taking less but suddenly it has increased
.
> >>>>>>
> >>>>>> Currently we do not have any monitoring  tool.
> >>>>>>
> >>>>>> On Tue, Aug 9, 2016 at 7:00 PM, Emir Arnautovic <
> >>>>>> emir.arnautovic@sematext.com> wrote:
> >>>>>>
> >>>>>> Hi Midas,
> >>>>>>>
> >>>>>>> Can you give us more details on your index: size, number
of new
> docs
> >>>>>>> between commits. Why do you think 1.3s for commit is to
much and
> why
> >>>>>>> do
> >>>>>>> you
> >>>>>>> need it to take less? Did you do any system/Solr monitoring?
> >>>>>>>
> >>>>>>> Emir
> >>>>>>>
> >>>>>>>
> >>>>>>> On 09.08.2016 14:10, Midas A wrote:
> >>>>>>>
> >>>>>>> please reply it is urgent.
> >>>>>>>>
> >>>>>>>> On Tue, Aug 9, 2016 at 11:17 AM, Midas A <test.midas@gmail.com>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi ,
> >>>>>>>>
> >>>>>>>>> commit is taking more than 1300 ms . what should
i check on
> server.
> >>>>>>>>>
> >>>>>>>>> below is my configuration .
> >>>>>>>>>
> >>>>>>>>> <autoCommit> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> <
> >>>>>>>>> openSearcher>false</openSearcher> </autoCommit>
<autoSoftCommit>
> >>>>>>>>> <maxTime>
> >>>>>>>>> ${solr.autoSoftCommit.maxTime:-1}</maxTime>
</autoSoftCommit>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>
> >>>>>>> Monitoring * Alerting * Anomaly Detection * Centralized
Log
> Management
> >>>>>>> Solr & Elasticsearch Support * http://sematext.com/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>
> >> --
> >> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> >> Solr & Elasticsearch Support * http://sematext.com/
> >>
>
>


-- 
--------------------------

Benedetti Alessandro
Visiting card : http://about.me/alessandro_benedetti

"Tyger, tyger burning bright
In the forests of the night,
What immortal hand or eye
Could frame thy fearful symmetry?"

William Blake - Songs of Experience -1794 England

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message