lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Luis Cappa Banda <luisca...@gmail.com>
Subject Re: Solr 4.1: problems with Spatial Search.
Date Sat, 09 Mar 2013 17:03:39 GMT
Hello again!

Uhm, so if I have understood then if I´m writing/reading from the index at
the same time (in other words, indexing operations are executing at the
same that that other ones are querying) the performance goes down, isn´t it?

Refering to the fieldType that I´m using, I´m just testing and I just want
to implement the classic scenario where each document has one coordinate
and to which execute geo spatial queries "given a circle with X radius". I
was testing with a multiValued field because I was wondering to include one
or more coordinates for each document due to the requirements of the
project that I´m developing: I´m tracking users positions until a top of N
positions (coordinates) to be stored in that multiValued field, but I can
adapt into just one coordinate per user if that change gets a better
performance.

In other words, David, and to summary:

- I just one to implement the classic scenario "coordinate + circle +
radius X".

- Due to NRT user position changes, I will write & read into the index
continuosly.

- I was testing with that fieldType multiValued, but I can change it into
another fieldType and into a single value field type if that improves
performance. In that case, what would you recommend?

- If even changing fieldTypes, multiValues or not, etc. the performance
doesn´t improves, what other alternatives would you recommend me to use?
I´m a Solr lover from Solr 1.4 version, but I´m familiar to other Spatial
Search alternatives technologies/tools/dbs like MongoDB. Any suggestions or
recommendations?



Best regards,

- Luis Cappa

2013/3/6 David Smiley (@MITRE.org) <DSMILEY@mitre.org>

> Luis,
> I should have asked how much data you have when I offered the solution.
>
> If you have a multi-valued spatial field and you need to get the closest
> of potentially many indexed points (and your schema snippet below shows
> multiValued=true) then I'm afraid you're stuck with this until the
> underlying distance caching mechanism is improved.  This is the biggest
> limitation of this field type.  See this issue for background on the
> problem:
> https://issues.apache.org/jira/browse/LUCENE-4698
> I suggest "watching" that issue to be notified of changes.  I've got a
> couple approaches to a solution on the horizon but it unfortunately hasn't
> been a priority for my time.
>
> RE 3-4 seconds… the first time it needs to build the cache (slow) but then
> speed-wise it shouldn't be too bad — it depends on how many documents you
> actually matched in your query, not how many might be in the system.  So
> instead please tell me how many documents your search matched (aka
> "numFound").  If you're doing simultaneous committing then this approach is
> completely un-workable.
>
> ~ David
>
> From: "Rakudten [via Lucene]" <ml-node+s472066n4045273h53@n3.nabble.com
> <mailto:ml-node+s472066n4045273h53@n3.nabble.com>>
> Date: Wednesday, March 6, 2013 11:48 AM
> To: "Smiley, David W." <dsmiley@mitre.org<mailto:dsmiley@mitre.org>>
> Subject: Re: Solr 4.1: problems with Spatial Search.
>
> I´ve doing some performance tests and I´ve noticed that with the new query
> syntax that David told me to use the QTime increases a lot. I´ve got an
> index with up to 8 million docs and sometimes the query response delays
> three, four or more seconds until finishes. Is that normal?
>
> 2013/3/6 Luis Cappa Banda <[hidden
> email]</user/SendEmail.jtp?type=node&node=4045273&i=0>>
>
> > Hey David, it works! Thank you very much. The true is that the
> > docummentation is a little bit confusing, but know It works perfectly.
> >
> > Regards,
> >
> > - Luis Cappa
> >
> > 2013/3/6 David Smiley (@MITRE.org) <[hidden
> email]</user/SendEmail.jtp?type=node&node=4045273&i=1>>
> >
> > Ah; bingo!
> >>
> >> The top error in the log is what Solr reports in the HTTP response you
> >> reported but it's the message of the exception wrapped by it in the logs
> >> which is more indicative of the problem:
> >>
> >> Caused by: org.apache.solr.common.SolrException: A ValueSource isn't
> >> directly available from this field. Instead try a query using the
> distance
> >> as the score.
> >>
> >> That error message (which I wrote) even contains the solution :-)
> >>
> >> You're using geodist() against solr.SpatialRecursivePrefixTreeFieldType
> >> which isn't supported.  You can get the distance but not using that
> >> approach.  Instead the query itself returns the distance as the score.
>  In
> >> the example schema you'll see a link to documentation about this field
> >> type
> >> which is this URL:
> >> http://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4
> >> From there click on "Sorting and Relevancy":
> >>
> >>
> http://wiki.apache.org/solr/SolrAdaptersForLuceneSpatial4#Sorting_and_Relevancy
> >> And you'll see an example query along the lines of what you want:
> >> &fl=*,score&sort=score asc&q={!
> >> score=distance}geo:"Intersects(Circle(54.729696,-98.525391 d=10))"
> >> (the score is the distance in this case)
> >>
> >> ~ David
> >>
> >>
> >> Rakudten wrote
> >> > Hello everyone!
> >> >
> >> >    - I´m using Solr 4.1.0.
> >> >
> >> >
> >> >    - Yes, without the sort the query works perfectly.
> >> >
> >> >
> >> >    - The fieldType is as follows:
> >> >
> >> >
> >> > <field name="geolocation" type="location_rpt" indexed="true"
> >> > stored="true" multiValued="true" />
> >> > *    *
> >> >
> >> > <fieldType name="location_rpt"
> >> > class="solr.SpatialRecursivePrefixTreeFieldType"
> >> >
> >> >
> >> >
> >>
> spatialContextFactory="com.spatial4j.core.context.jts.JtsSpatialContextFactory"
> >> >                distErrPct="0.025"
> >> >                maxDistErr="0.000009"
> >> >                units="degrees"
> >> >          />
> >> >    - The error trace from Solr:
> >> >
> >> > Mar 6, 2013 10:02:29 AM org.apache.solr.common.SolrException log
> >> > SEVERE: org.apache.solr.common.SolrException: sort param could not be
> >> > parsed as a query, and is not a field that exists in the index:
> >> geodist()
> >> > at
> org.apache.solr.search.QueryParsing.parseSort(QueryParsing.java:340)
> >> > at org.apache.solr.search.QParser.getSort(QParser.java:281)
> >> > at
> >> >
> >>
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:123)
> >> > at
> >> >
> >>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:187)
> >> > at
> >> >
> >>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> >> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1816)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:448)
> >> > at
> >> >
> >>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:269)
> >> > at
> >> >
> >>
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
> >> > at
> >> >
> >>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:225)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
> >> > at
> >> >
> >>
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
> >> > at
> >> >
> >>
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
> >> > at
> >> >
> >>
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
> >> > at
> >> >
> >>
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
> >> > at
> >> >
> >>
> org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1001)
> >> > at
> >> >
> >>
> org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:585)
> >> > at
> >> >
> >>
> org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:312)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> >> > at java.lang.Thread.run(Thread.java:662)
> >> > Caused by: org.apache.solr.common.SolrException: A ValueSource isn't
> >> > directly available from this field. Instead try a query using the
> >> distance
> >> > as the score.
> >> > at
> >> >
> >>
> org.apache.solr.schema.AbstractSpatialFieldType.getValueSource(AbstractSpatialFieldType.java:219)
> >> > at
> >> >
> >>
> org.apache.solr.search.function.distance.HaversineConstFunction.parseSfield(HaversineConstFunction.java:168)
> >> > at
> >> >
> >>
> org.apache.solr.search.function.distance.HaversineConstFunction.access$200(HaversineConstFunction.java:47)
> >> > at
> >> >
> >>
> org.apache.solr.search.function.distance.HaversineConstFunction$1.parse(HaversineConstFunction.java:104)
> >> > at
> >> >
> >>
> org.apache.solr.search.FunctionQParser.parseValueSource(FunctionQParser.java:352)
> >> > at
> org.apache.solr.search.FunctionQParser.parse(FunctionQParser.java:68)
> >> > at org.apache.solr.search.QParser.getQuery(QParser.java:142)
> >> > at
> org.apache.solr.search.QueryParsing.parseSort(QueryParsing.java:271)
> >> > ... 22 more
> >> >
> >> >
> >> > I would like to sort by distance and also to return that distance as a
> >> fl
> >> > parameter inside the response.
> >> >
> >> > Thank you very much.
> >> >
> >> > Regards,
> >> >
> >> >
> >> > - Luis Cappa
> >> >
> >> >
> >> > 2013/3/5 Chris Hostetter &lt;
> >>
> >> > hossman_lucene@
> >>
> >> > &gt;
> >> >
> >> >>
> >> >> 1) which version of solr are you using?
> >> >> 2) what is the field & fieldtype for "geolocation"
> >> >> 2) can you try changing your query to "q={!func}geodist()" to verify
> >> that
> >> >> the function works at all?
> >> >>
> >> >>
> >> >>
> >> >> : Date: Tue, 5 Mar 2013 19:30:33 +0100
> >> >> : From: Luis Cappa Banda &lt;
> >>
> >> > luiscappa@
> >>
> >> > &gt;
> >> >> : Reply-To:
> >>
> >> > solr-user@.apache
> >>
> >> >> : To:
> >>
> >> > solr-user@.apache
> >>
> >> >> : Subject: Re: Solr 4.1: problems with Spatial Search.
> >> >> :
> >> >> : Without the sort it works perfectly, and there are no more error
> >> >> messages,
> >> >> : just the one I copy&pasted, :-(
> >> >> : El 05/03/2013 19:05, "David Smiley (@MITRE.org)" &lt;
> >>
> >> > DSMILEY@
> >>
> >> > &gt;
> >> >> : escribió:
> >> >> :
> >> >> : > Hmm; weird.  It looks right.  Does it work without the sort?
--
> >> i.e.
> >> >> does
> >> >> : > the
> >> >> : > filter work?  Are there more interesting looking error messages
> >> >> output
> >> >> by
> >> >> : > Solr?
> >> >> : >
> >> >> : >
> >> >> : > Rakudten wrote
> >> >> : > > Hello!
> >> >> : > >
> >> >> : > > I´m trying to sort by geodist() distance, but it seems
that I
> >> >> can´t:
> >> >> : > >
> >> >> : > > *The query:*
> >> >> : > >
> >> >> : > >
> >> >> : >
> >> >>
> >>
> http://192.168.1.129:8080/geo/zones/select?q=*:*&fq={!geofilt}&sfield=geolocation&pt=38.96442,-3.89047&d=10&wt=json&start=0&rows=10&sort=geodist()%20asc
> <
> http://192.168.1.129:8080/geo/zones/select?q=*:*&fq=%7B!geofilt%7D&sfield=geolocation&pt=38.96442,-3.89047&d=10&wt=json&start=0&rows=10&sort=geodist()%20asc
> >
> >> >> : > >
> >> >> : > > *The response:*
> >> >> : > > *
> >> >> : > > *
> >> >> : > >
> >> >> : > > {
> >> >> : > > responseHeader: {
> >> >> : > > status: 400,
> >> >> : > > QTime: 2,
> >> >> : > > params: {
> >> >> : > > d: "10",
> >> >> : > > sort: "geodist() asc",
> >> >> : > > start: "0",
> >> >> : > > q: "*:*",
> >> >> : > > sfield: "geolocation",
> >> >> : > > pt: "38.96442,-3.89047",
> >> >> : > > wt: "json",
> >> >> : > > fq: "{!geofilt}",
> >> >> : > > rows: "10"
> >> >> : > > }
> >> >> : > > },
> >> >> : > > error: {
> >> >> : > > msg: "sort param could not be parsed as a query, and is
not a
> >> field
> >> >> that
> >> >> : > > exists in the index: geodist()",
> >> >> : > > code: 400
> >> >> : > > }
> >> >> : > > }
> >> >> : > >
> >> >> : > > I´ve read several times the documentation and it seems
that
> this
> >> is
> >> >> the
> >> >> : > > correct way to search filtering by coordinates and sorting
by
> >> >> distance,
> >> >> : > > but
> >> >> : > > I doesn´t work. Any suggestions?
> >> >> : > >
> >> >> : > > Also I´ve tried to return the distance as a field via
fl
> params,
> >> >> but
> >> >> I
> >> >> : > > couldn´t. It may help me a lot if anyone could tell me
how to
> do
> >> >> it.
> >> >> : > >
> >> >> : > > Thank you very much in advance!
> >> >> : > >
> >> >> : > >
> >> >> : > > Best regards,
> >> >> : > >
> >> >> : > >
> >> >> : > > - Luis Cappa
> >> >> : >
> >> >> : >
> >> >> : >
> >> >> : >
> >> >> : >
> >> >> : > -----
> >> >> : >  Author:
> >> >> : >
> >> http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> >> >> : > --
> >> >> : > View this message in context:
> >> >> : >
> >> >>
> >>
> http://lucene.472066.n3.nabble.com/Solr-4-1-problems-with-Spatial-Search-tp4044868p4044958.html
> >> >> : > Sent from the Solr - User mailing list archive at Nabble.com.
> >> >> : >
> >> >> :
> >> >>
> >> >> -Hoss
> >>
> >>
> >>
> >>
> >>
> >> -----
> >>  Author:
> >> http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> >> --
> >> View this message in context:
> >>
> http://lucene.472066.n3.nabble.com/Solr-4-1-problems-with-Spatial-Search-tp4044868p4045231.html
> >> Sent from the Solr - User mailing list archive at Nabble.com.
> >>
> >
> >
>
>
> ________________________________
> If you reply to this email, your message will be added to the discussion
> below:
>
> http://lucene.472066.n3.nabble.com/Solr-4-1-problems-with-Spatial-Search-tp4044868p4045273.html
> To unsubscribe from Solr 4.1: problems with Spatial Search., click here<
> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=4044868&code=RFNNSUxFWUBtaXRyZS5vcmd8NDA0NDg2OHwxMDE2NDI2OTUw
> >.
> NAML<
> http://lucene.472066.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml
> >
>
>
>
>
> -----
>  Author:
> http://www.packtpub.com/apache-solr-3-enterprise-search-server/book
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Solr-4-1-problems-with-Spatial-Search-tp4044868p4045295.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message