lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexey Serba <ase...@gmail.com>
Subject Re: Re: About Solr DataImportHandler
Date Wed, 08 Sep 2010 05:50:31 GMT
> 2.But there are some problems:
> if the table is very big,solr will spend a long time to import and index,may
> be one day and more.so once occurred network problems and others during this
> time,maybe solr can not remember what documents had processed,and if we
> continue data import ,we do not know where to start.

You can _batch_ import your data using full import command by
providing additional request parameter ( see
http://wiki.apache.org/solr/DataImportHandler#Accessing_request_parameters
), i.e.

query="SELECT * FROM my_table ORDER BY id LIMIT 1000000 OFFSET
${dataimporter.request.offset}"

and then calling full-import command several times:
1) /dataimport?clean=true&offset=0
2) /dataimport?clean=false&offset=1000000
3) /dataimport?clean=false&offset=2000000
etc

// Please use solr-user@lucene.apache.org mailing list for such
questions. _dev_ is not appropriate place for this.

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message