lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adam Neal [Extranet]" <AN...@mass.co.uk>
Subject RE: Slow commits
Date Mon, 22 Feb 2016 09:37:40 GMT
Well I got the numbers wrong, there are actually around 66000 fields on the index. I have restructured
the index and there are now around 1500 fiields. This has resulted in the commit taking 34
seconds which is acceptable for my usage however it is still significantly slower than the
4.10.2 commit on the original 66000 fields which was around 1 second.
________________________________________
From: Adam Neal [Extranet] [ANeal@mass.co.uk]
Sent: 19 February 2016 17:43
To: solr-user@lucene.apache.org
Subject: RE: Slow commits

I'm out of the office now so I don't have the numbers to hand but from memory I think there
are probably around 800-1000 fields or so. I will confirm on Monday.

If i have time over the weekend I will try and recreate the problem at home and see if I can
post up a sample.
________________________________________
From: Yonik Seeley [yseeley@gmail.com]
Sent: 19 February 2016 16:25
To: solr-user@lucene.apache.org
Subject: Re: Slow commits

On Fri, Feb 19, 2016 at 8:51 AM, Adam Neal [Extranet] <ANeal@mass.co.uk> wrote:
> I've recently upgraded from 4.10.2 to 5.3.1 and I've hit an issue with slow commits on
one of my cores. The core in question is relatively small (56k docs) and the issue only shows
when commiting after a number of deletes, commiting after additions is fine. As an example
commiting after deleting approximately 10% of the documents takes around 25mins. The same
test on the 4.10.2 instance takes around 1 second.
>
> I have done some investigation and the problem appears to be caused by having dynamic
fields, the core in question has a large number, performing the same operation on this core
with the dynamic fields removed sees a big improvement on the performance with the commit
taking 11 seconds (still not quite on a par with 4.10.2).

Dynamic fields is a Solr schema concept, and does not translate to any
differences in Lucene.
You may be hitting something due to a large number of fields (at the
lucene level, each field name is a different field).  How many
different fields (i.e. fieldnames) do you have across the entire
index?

-Yonik

#####################################################################################

This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only
for the use of the addressee or with its permission.  Use by anyone else for any purpose is
prohibited.  If you are not the addressee, you should not use, disclose, copy or distribute
this e-mail and should notify us of receipt immediately by return e-mail to the address where
the e-mail originated.

This E-mail may not have been sent through a secure system and accordingly (i) its contents
should not be relied upon by any person without independent verification from Mass Consultants
Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission,
opening or use of this message and any attachments will not adversely affect its systems or
data. No responsibility is accepted by Mass Consultants Ltd in this regard.

Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems
or persons other than the addressee, for the purposes of ascertaining whether the communication
complies with the law and Mass Consultants Ltd's policies.

Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road,
Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600.

#####################################################################################

#####################################################################################

This E-mail is the property of Mass Consultants Ltd. It is confidential and intended only
for the use of the addressee or with its permission.  Use by anyone else for any purpose is
prohibited.  If you are not the addressee, you should not use, disclose, copy or distribute
this e-mail and should notify us of receipt immediately by return e-mail to the address where
the e-mail originated.

This E-mail may not have been sent through a secure system and accordingly (i) its contents
should not be relied upon by any person without independent verification from Mass Consultants
Ltd and (ii) it is the responsibility of the recipient to ensure that the onward transmission,
opening or use of this message and any attachments will not adversely affect its systems or
data. No responsibility is accepted by Mass Consultants Ltd in this regard.

Any e-mails that are sent to Mass Consultants Ltd's e-mail addresses may be monitored by systems
or persons other than the addressee, for the purposes of ascertaining whether the communication
complies with the law and Mass Consultants Ltd's policies.

Mass Consultants Ltd is registered in England No. 1705804, Enterprise House, Great North Road,
Little Paxton, Cambs., PE19 6BN, United Kingdom. Tel: +44 (0) 1480 222600.

#####################################################################################

Mime
View raw message