incubator-couchdb-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sho Fukamachi <sho.fukama...@gmail.com>
Subject Re: using couchdb for 100GB geodata from openstreetmap
Date Mon, 23 Mar 2009 08:55:41 GMT

On 23/03/2009, at 4:58 PM, James Marca wrote:

> On Mon, Mar 23, 2009 at 12:07:46AM +0100, Volker Mische wrote:
>> Is there a difference between a custom view engine and the approach I
>> try to pursue with GeoCouch? It still comes down to the fact that you
>> need to store the spatial information in some kind of (external)  
>> spatial
>> index.
>
> And at the risk of being completely wrong, I understood that a geo
> index isn't a B-tree (PostGIS uses gist indexes).  So that may be
> hard to implement as a view server.  But just now in checking up on
> this, I ran across a recent blog entry here
> http://www.rooftopsolutions.nl/article/231 that talked about using  
> morton
> numbers to index spatial data.  I have no clue on that beyond
> http://en.wikipedia.org/wiki/Morton_number_(number_theory), but it
> sounds pretty promising.

You might also want to look into Hilbert curves, a slightly different  
way of doing the same thing, but with more consistent "winding".  
Plenty of JS implementations floating about.

The problem with this kind of thing, of course, is that it all breaks  
down around the meridian - ie you're going to have to draw a line on  
earth somewhere where just east of that line is 1˚ lattitude and just  
west is -359, which totally screws up the "near numbers are local"  
idea. Dealing with this is half the reason the GIS systems exist.  
There's no real way around it but a quick and dirty kind-of "solution"  
is to offset all your data by around -170 lat so your "crossover" is  
in a relatively unpopulated area. If you use the unreformed data, 0 is  
smack bang in the middle of London, which might cause a few problems ; )


> The Geo Couch approach, as I understand it (again, forgive any
> ignorance on my part) is an external process that responds to queries,
> right?  So there isn't an index that CouchDB understands, and you
> can't run map/reduce type queries over it.  At least, that's why I
> didn't try to load up GeoCouch.  I was looking for a view engine that
> was a bit more native to the map/reduce aspects of CouchDB.
>
> Maybe Morton numbers can provide that with just a regular javascript
> view??  On a super quick read, it generates a 64 bit integer that is
> similar for spatially proximate regions.  Maybe all you have to do is
> code up the Morton's number algorithm in your view???
>
> Regards,
> James
>>
>> Cheers,
>>  Volker
>>
>> Damien Katz wrote:
>>> Random idea: a custom view engine for geospatial data
>>>
>>> Store the documents like normal and have the custom engine index  
>>> them
>>> for geospatial queries. Then you query the view engine (GET
>>> /db/_geo?yada=... )
>>>
>>> -Damien
>>>
>>> On Mar 22, 2009, at 6:54 PM, Volker Mische wrote:
>>>
>>>> pablo platt wrote:
>>>>> On Mon, Mar 23, 2009 at 12:32 AM, Volker Mische
>>>>> <volker.mische@gmail.com>wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I'll only reply to the GeoCouch bit:
>>>>>>
>>>>>>> I've read about geocouch -
>>>>>>>
>>>>>> http://vmx.cx/cgi-bin/blog/index.cgi/geocouch-geospatial-queries-with-couchdb:2008-10-26:en,CouchDB,Python,geo
>>>>>>
>>>>>>> and my question is if a geo aware couchdb can be built without
 
>>>>>>> any
>>>>>>> python
>>>>>>> and SpatiaLite code?
>>>>>>>
>>>>>>> If I want to be able to make proximity searches - find banks
 
>>>>>>> in 3km
>>>>>>> range
>>>>>>> from st some-street
>>>>>>> do I need to create some kind of index docs for the views?
>>>>>> The problem is that proximity searches are kind of dynamic.  
>>>>>> CouchDB
>>>>>> views are like precomputed result-sets, but you can't  
>>>>>> precompute all
>>>>>> possible values, as the center of the search could possibly be
>>>>>> anywhere.
>>>>>> Therefore you need something that makes such dynamic requests  
>>>>>> possible.
>>>>>>
>>>>>> Pure CouchDB could be used if you constrain your searches. You  
>>>>>> would
>>>>>> need to restrict to things like, a certain number for the  
>>>>>> center of
>>>>>> searches, only a fixed number of radii, and a fixed number of  
>>>>>> location
>>>>>> types (like banks, restaurants etc).
>>>>>>
>>>>>> Cheers,
>>>>>> Volker
>>>>>>
>>>>>
>>>>> I've asked on the #couchdb channel and got the same answer.
>>>>>
>>>>> The 'Name Finder' on OSM
>>>>> http://wiki.openstreetmap.org/index.php/Name_finder
>>>>> divides the map to region and add a field to each element(place)
>>>>> indicating
>>>>> what region it belongs.
>>>>> When you make a search "banks near someRoad" it find in which  
>>>>> region
>>>>> someRoad is and find all the banks in that road.
>>>>> So you are limited to predefined distances but the above design  
>>>>> will
>>>>> work in
>>>>> couchdb as well.
>>>>> There is an assumption that they used this design because they  
>>>>> used
>>>>> MySQL
>>>>> but I can't verify that.
>>>>>
>>>>> My question is if arbitrary distances are not a requirement of my
>>>>> app, is it
>>>>> better to use couchdb for storing the geodata
>>>>> or is it better to use a spatial db like postGIS in any case.
>>>>>
>>>>> Thanks
>>>>
>>>> As much as I love CouchDB, your application sounds like a typical  
>>>> case
>>>> for PostGIS. The tools to import OSM data into PostGIS are there,  
>>>> and
>>>> you have a spatial index. Though you would obviously lose all the  
>>>> other
>>>> nice CouchDB features (like replication etc).
>>>>
>>>> Cheers,
>>>> Volker
>>>>
>>>>
>>>>
>>>
>
> -- 
> This message has been scanned for viruses and
> dangerous content by MailScanner, and is
> believed to be clean.
>


Mime
View raw message