lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Erik Hatcher <erik.hatc...@gmail.com>
Subject Re: Problem in indexing on large data set by Dataimporthandler in solr
Date Mon, 01 Feb 2010 13:53:29 GMT
Can you give it a shot on Solr 1.4 instead?  DIH has had numerous  
enhancements/fixes since 1.3.

	Erik


On Feb 1, 2010, at 8:42 AM, Vijayant Kumar wrote:

>
>> Hi,
>>
>> I am trying to index some large set of data in solr using
>> dataimporthandler.
>>
>> It is working fine for small set but when I am trying to index on  
>> large
>> set it produces error.
>>
>> I am using solr version 1.3 and mysql version  Ver 14.7 Distrib  
>> 4.1.20,
>> for
>> redhat-linux-gnu (i686)
>>
>>
>> Earlier I was facing java heap space error
>>
>> Exception in thread "Thread-16" java.lang.OutOfMemoryError: Java heap
>> space
>>
>> Get it solve by allocating java -Xmx512M -Xms512M -jar start.jar
>>
>> But Now it is giving mysql communication link failure error
>>
>> com.mysql.jdbc.CommunicationsException: Communications link failure  
>> due to
>> underlying exception:
>>
>> ** BEGIN NESTED EXCEPTION **
>>
>> java.io.EOFException
>>
>> STACKTRACE:
>>
>> java.io.EOFException
>>        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1913)
>> I had added transactionIsolation="TRANSACTION_READ_COMMITTED"
>> holdability="CLOSE_CURSORS_AT_COMMIT" in this schema as suggested  
>> by solr
>> wiki page
>> but it is still not working.
>>
>> Here is my data-config.xml file
>>
>> <?xml version="1.0" encoding="UTF-8" ?>
>> <dataConfig>
>> <dataSource name="jdbc" driver="com.mysql.jdbc.Driver"
>> url="jdbc:mysql://localhost/databasename"
>> user="xxx" password="****" batchSize="-1" readOnly="true"
>> autoCommit="false" transactionIsolation="TRANSACTION_READ_COMMITTED"
>> holdability="CLOSE_CURSORS_AT_COMMIT"/>
>> <document name="feature">
>> <entity name="posts" dataSource="jdbc" pk="pid" query="
>> SELECT uid, pid, message FROM xxx"
>> transformer="TemplateTransformer">
>> <add>
>> <doc>
>> <field column = "pid" name="pid"/>
>> <field column = "uid" name="uid"/>
>> <field column = "message" name="message"/>
>> </doc>
>> </add>
>> </entity>
>> </document>
>> </dataConfig>
>>
>>
>>
>> My table contain around 1.5 core of data on local machine On  
>> production it
>> is
>> containing around 4 times of data as on local.
>>
>>
>>
>> Thank you,
>> Vijayant Kumar
>> Software Engineer
>> Website Toolbox Inc.
>> http://www.websitetoolbox.com
>> 1-800-921-7803 x211
>>
>
>
> -- 
>
> Thank you,
> Vijayant Kumar
> Software Engineer
> Website Toolbox Inc.
> http://www.websitetoolbox.com
> 1-800-921-7803 x211
>


Mime
View raw message