lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin Freas <cfr...@stsci.edu>
Subject Re: Using Solr's spatial functionality for astronomical catalog
Date Tue, 16 Feb 2016 08:31:47 GMT

David, thanks for getting back to me.  SpatialRecursivePrefixTreeFieldType
seems to be what I need, and the default search seems appropriate.  This
is for entries in an astronomical catalog, so great circle distances on a
perfect sphere is what I¹m after.

I am having a bit of difficulty though.

Having gotten records importing via our database into a schema on both a
stand-along Solr instance and in a SolrCloud cluster, I¹ve moved on to
³spatializing² the appropriate fields, and everything looks like it¹s
working, in that there are no errors thrown.  But when I try what I think
is valid spatial query, it doesn¹t work.

Here¹s what I¹m doing.  Pertinent bits from my schema:

	...
		<field name="SourceRpt" type="rpt" indexed="true" stored="false"
required="false" multiValued="false" />
		<uniqueKey>CatID</uniqueKey>
	</fields>
	<types>
		<fieldType name="rpt" class="solr.SpatialRecursivePrefixTreeFieldType"
			geo="true"
			distanceUnits="degrees" />
	...
	</types>

In my db-config.xml, I¹ve got this sql:
	<entity name="observation" query="SELECT CatID, MatchRA, MatchDec,
SourceRA, SourceDec,  '''' + ltrim(rtrim(Str(SourceRA,25,16))) + ',' +
ltrim(rtrim(Str(SourceDec,25,16))) + '''' FROM dbo.Catalog" />


When I run a data import through Solr¹s admin gui and look at the verbose
debug output, something seems off.  Top of the output is this:
	{
	"responseHeader": {
	"status": 0,
	"QTime": 75
	},
	"initArgs": [
	"defaults",
	[
	"config",
	"hsc-db-config.xml"
	]
	],
	"command": "full-import",
	"mode": "debug",
	"documents": [
	{
	"MatchDec": [
	-0.673125699999921
	],
	"SourceDec": [
	-0.673125699999921
	],
	"MatchRA": [
	0.5681586795334927
	],
	"SourceRA": [
	0.5681586795334927
	],
	"CatID": [
	25558943
	]
	},


There¹s no SourceRpt field there.  But in verbose output of what¹s
returned from the query, the SourceRpt field seems to be correctly put
together:

	"verbose-output": [
	"entity:observation",
	[
	"document#1",
	[
	"query",
	"SELECT CatID, MatchRA, MatchDec, SourceRA, SourceDec,  '''' +
ltrim(rtrim(Str(SourceRA,25,16))) + ',' +
ltrim(rtrim(Str(SourceDec,25,16))) + '''' FROM xcat.BestCatalog",
	"time-taken",
	"0:0:0.22",
	null,
	"----------- row #1-------------",
	"",
	"'0.5681586795334928,-0.6731256999999210'",
	"MatchDec",
	-0.673125699999921,
	"SourceDec",
	-0.673125699999921,
	"MatchRA",
	0.5681586795334927,
	"SourceRA",
	0.5681586795334927,
	"CatID",
	25558943,
	null,
	"---------------------------------------------"
	]



I try a spatial search like this:
	http://localhost:8983/solr/spatial/select?q=*%3A*&wt=json&indent=true&spat
ial=true&pt=3%2C3&sfield=SourceRpt&d=0.0001


... And I get back all (10) records in the core, when I would expect 0,
given the very small distance I supply to a point well away from any of
the records.

I¹m not sure what¹s going on.  I don¹t know if this is a simple Solr
config error I¹m missing, or if there¹s some spatial magic I¹m unaware of.
 

Any thoughts appreciated.

-Colin


On 1/20/16, 9:34 PM, "david.w.smiley@gmail.com" <david.w.smiley@gmail.com>
wrote:

>Hello Colin,
>
>If the spatial field you use is the SpatialRecursivePrefixTreeFieldType
>one
>(RPT for short) with geo="true" then the circle shape (i.e. point-radius
>filter) implied by the geofilt Solr QParser is on a sphere.  That is, it
>uses the "great circle" distance computed using the Haversine formula by
>default, though it can be configured to use the Law of Cosines formula or
>Vincenty (spherical version) formula if you so choose.  Using geodist()
>for
>spatial distance sorting/boosting also uses this.  If you use LatLonType
>then geofilt & geodist() use Haversine too.
>
>If you use polygons or line strings, then it's *not* using a spherical
>model; it's using a Euclidean (flat) model on plate carrée.  I am
>currently
>working on adapting the Spatial4j library to work with Lucene's Geo3D (aka
>spatial 3d) which has both a spherical model and an ellipsoidal model,
>which can be configured with the characteristics specified by WGS84.  If
>you are super-eager to get this yourself without waiting, then you could
>write a Solr QParser that constructs a Geo3dShape wrapping a Geo3D
>GeoShape
>object constructed from query parameters.  You might alternatively try and
>use Geo3DPointField on Lucene 6 trunk.
>
>~ David
>
>On Tue, Jan 19, 2016 at 11:07 AM Colin Freas <cfreas@stsci.edu> wrote:
>
>>
>> Greetings!
>>
>> I have recently stood up an instance of Solr, indexing a catalog of
>>about
>> 100M records representing points on the celestial sphere.  All of the
>> fields are strings, floats, and non-spatial types.  I¹d like to convert
>>the
>> positional data to an appropriate spatial point data type supported by
>>Solr.
>>
>> I have a couple of questions about indexing spatial data using Solr,
>>since
>> it seems spatial4j, and the spatial functionality in Solr generally, is
>> more GIS geared.  I worry that the measurements of lat/long on the
>> imperfect sphere of the Earth wouldn¹t match up with the astronomical
>>right
>> ascension/declination concept of the perfectly spherical celestial
>>sphere
>> used to record the coordinates of our records.
>>
>> I¹m also worried there might be other assumptions built into spatial4j &
>> Solr based on using a real surface vs a virtual one.
>>
>> Does anyone have experience doing this, or is there perhaps some
>> documentation specific to this use case that anyone might be able to
>>point
>> me to?
>>
>> Thanks in advance,
>> Colin
>>
>-- 
>Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
>LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
>http://www.solrenterprisesearchserver.com


Mime
View raw message