lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From eks dev <eks...@yahoo.co.uk>
Subject Re: Unusually long data import time?
Date Wed, 22 Feb 2012 18:55:04 GMT
Davon, you ought to try to update from many threads, (I do not know if
DIH can do it, check it), but lucene does great job if fed from many
update threads...

depends where your time gets lost, but it is usually a) analysis chain
or b) database

if it os a) and your server has spare cpu-cores, you can scale at X
NooCores rate

On Wed, Feb 22, 2012 at 7:41 PM, Devon Baumgarten
<dbaumgarten@nationalcorp.com> wrote:
> Ahmet,
>
> I do not. I commented autoCommit out.
>
> Devon Baumgarten
>
>
>
> -----Original Message-----
> From: Ahmet Arslan [mailto:iorixxx@yahoo.com]
> Sent: Wednesday, February 22, 2012 12:25 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Unusually long data import time?
>
>> Would it be unusual for an import of 160 million documents
>> to take 18 hours?  Each document is less than 1kb and I
>> have the DataImportHandler using the jdbc driver to connect
>> to SQL Server 2008. The full-import query calls a stored
>> procedure that contains only a select from my target table.
>>
>> Is there any way I can speed this up? I saw recently someone
>> on this list suggested a new user could get all their Solr
>> data imported in under an hour. I sure hope that's true!
>
> Do have autoCommit or autoSoftCommit configured in solrconfig.xml?

Mime
View raw message