lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Stuart <david.stu...@progressivealliance.co.uk>
Subject Re: Importing large datasets
Date Wed, 02 Jun 2010 19:00:02 GMT
How long does it take to do a grab of all the data via SQL? I found by  
denormalizing the data into a lookup table meant that I was able to  
index about 300k rows of similar data size with dih regex spilting on  
some fields in about 8mins I know it's not quite the scale bit with  
batching...

David Stuar

On 2 Jun 2010, at 17:58, Blargy <zmanods@hotmail.com> wrote:

>
>
>
>> One thing that might help indexing speed - create a *single* SQL  
>> query
>> to grab all the data you need without using DIH's sub-entities, at
>> least the non-cached ones.
>>
>
> Not sure how much that would help. As I mentioned that without the  
> item
> description import the full process takes 4 hours which is bearable.  
> However
> once I started to import the item description which is located on a  
> separate
> machine/database the import process exploded to over 24 hours.
>
> -- 
> View this message in context: http://lucene.472066.n3.nabble.com/Importing-large-datasets-tp863447p865324.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message