lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "O. Olson" <olson_...@yahoo.it>
Subject DataImport using SqlEntityProcessor running Out of Memory
Date Fri, 09 May 2014 15:16:15 GMT
I have a Data Schema which is Hierarchical i.e. I have an Entity and a number
of attributes. For a small subset of the Data - about 300 MB, I can do the
import with 3 GB memory. Now with the entire 4 GB Dataset, I find I cannot
do the import with 9 GB of memory. 
I am using the SqlEntityProcessor as below: 

<dataConfig>
    <dataSource driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
url="jdbc:sqlserver://localhost\MSSQLSERVER;databaseName=SolrDB;user=solrusr;password=solrusr;"/>
    <document>
			<entity name="Entity" query="SELECT EntID, Image FROM ENTITY_TABLE">
				<field column="EntID" name="EntID" />
				<field column="Image" name="Image" />
				
				<entity name="EntityAttribute1"  
                    query="SELECT AttributeValue, EntID FROM ATTR_TABLE
WHERE AttributeID=1" 
					cacheKey="EntID" cacheLookup="Entity.EntID"
processor="SqlEntityProcessor" cacheImpl="SortedMapBackedCache">
					<field column="AttributeValue" name="EntityAttribute1" /> 
				</entity>
				<entity name="EntityAttribute2"  
                    query="SELECT AttributeValue, EntID FROM ATTR_TABLE
WHERE AttributeID=2" 
					cacheKey="EntID" cacheLookup="Entity.EntID"
processor="SqlEntityProcessor" cacheImpl="SortedMapBackedCache">
					<field column="AttributeValue" name="EntityAttribute2" /> 
				</entity>
				
				
				
	        </entity>
    </document>
</dataConfig>



What is the best way to import this data? Doing it without a cache, results
in many SQL queries. With the cache, I run out of memory. 

I’m curious why 4GB of data cannot entirely fit in memory. One thing I need
to mention is that I have about 400 to 500 attributes. 

Thanks in advance for any helpful advice. 
O. O. 




--
View this message in context: http://lucene.472066.n3.nabble.com/DataImport-using-SqlEntityProcessor-running-Out-of-Memory-tp4135080.html
Sent from the Solr - User mailing list archive at Nabble.com.

Mime
View raw message