lucene-solr-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Solr Wiki] Update of "DataImportHandlerFaq" by NoblePaul
Date Wed, 19 Nov 2008 04:26:33 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Solr Wiki" for change notification.

The following page has been changed by NoblePaul:
http://wiki.apache.org/solr/DataImportHandlerFaq

------------------------------------------------------------------------------
-  * I'm using DataImportHandler with a MySQL database. My table is huge and DataImportHandler
is going out of memory. Why does DataImportHandler bring everything to memory?
+ === I'm using DataImportHandler with a MySQL database. My table is huge and DataImportHandler
is going out of memory. Why does DataImportHandler bring everything to memory? ===
-   * DataImportHandler is designed to stream row one-by-one. It passes a fetch size value
(default: 500) to Statement#setFetchSize which some drivers do not honor. For MySQL, add batchSize
property to dataSource configuration with value -1. This will pass Integer.MIN_VALUE to the
driver as the fetch size and keep it from going out of memory for large tables.
+ DataImportHandler is designed to stream row one-by-one. It passes a fetch size value (default:
500) to Statement#setFetchSize which some drivers do not honor. For MySQL, add batchSize property
to dataSource configuration with value -1. This will pass Integer.MIN_VALUE to the driver
as the fetch size and keep it from going out of memory for large tables.
  
-  * I'm using DataImportHandler with MS SQL Server database with sqljdbc driver. DataImportHandler
is going out of memory. I tried adjustng the batchSize values but they don't seem to make
any difference. How do I fix this?
+ === I'm using DataImportHandler with MS SQL Server database with sqljdbc driver. DataImportHandler
is going out of memory. I tried adjustng the batchSize values but they don't seem to make
any difference. How do I fix this? ===
-   * There's a connection property called responseBuffering in the sqljdbc driver whose default
value is "full" which causes the entire result set to be fetched. See http://msdn.microsoft.com/en-us/library/ms378988.aspx
for more details. You can set this property to "adaptive" to keep the driver from getting
everything into memory. Connection properties like this can be set as an attribute (responseBuffering="adaptive")
in the dataSource configuration OR directly in the jdbc url specified in DataImportHandler's
dataSource configuration.
+ There's a connection property called responseBuffering in the sqljdbc driver whose default
value is "full" which causes the entire result set to be fetched. See http://msdn.microsoft.com/en-us/library/ms378988.aspx
for more details. You can set this property to "adaptive" to keep the driver from getting
everything into memory. Connection properties like this can be set as an attribute (responseBuffering="adaptive")
in the dataSource configuration OR directly in the jdbc url specified in DataImportHandler's
dataSource configuration.
-  * Is it possible to use core properties inside data-config xml?  
-   * Add your property in the invariant section of solrconfig's DataImportHandler element.
For example, add this section:
-   * <lst name="invariants"> <str name="xmlDataDir">${xmlDataDir}</str>
</lst>
-   * Use it as ${dataimporter.request.xmlDataDir} in your data-config to access this. 
-   * [http://markmail.org/message/svlmsygl3sss4f67 see the mail thread]
  
+ === Is it possible to use core properties inside data-config xml? ===
+  
+ 
+  * Add your property in the invariant section of solrconfig's DataImportHandler element.
For example, add this section:
+  * <lst name="invariants"> <str name="xmlDataDir">${xmlDataDir}</str>
</lst>
+  * Use it as ${dataimporter.request.xmlDataDir} in your data-config to access this. 
+  * [http://markmail.org/message/svlmsygl3sss4f67 see the mail thread]
+ 

Mime
View raw message