lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Shawn Heisey <s...@elyograg.org>
Subject Re: Out of Memory when i downdload 5 Million records from sqlserver to solr
Date Thu, 03 Jul 2014 04:07:41 GMT
On 7/1/2014 4:57 AM, mskeerthi wrote:
> I have to download my 5 million records from sqlserver to solr into one
> index. I am getting below exception after downloading 1 Million records. Is
> there any configuration or another to download from sqlserver to solr.
> 
> Below is the exception i am getting in solr:
> org.apache.solr.common.SolrException; auto commit
> error...:java.lang.IllegalStateException: this writer hit an
> OutOfMemoryError; cannot commit

JDBC has a bad habit of defaulting to a mode where it will try to load
the entire SQL result set into RAM.  Different JDBC drivers have
different ways of dealing with this problem.  For Microsoft SQL Server,
here's a guide:

https://wiki.apache.org/solr/DataImportHandlerFaq#I.27m_using_DataImportHandler_with_MS_SQL_Server_database_with_sqljdbc_driver._DataImportHandler_is_going_out_of_memory._I_tried_adjustng_the_batchSize_values_but_they_don.27t_seem_to_make_any_difference._How_do_I_fix_this.3F

If you have trouble with that really long URL in your mail client, just
visit the main FAQ page and click on the link for SQL Server:

https://wiki.apache.org/solr/DataImportHandlerFaq

Thanks,
Shawn


Mime
View raw message