lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Noble Paul നോബിള്‍ नोब्ळ् <noble.p...@gmail.com>
Subject Re: DataImporter : Java heap space
Date Tue, 14 Apr 2009 05:47:43 GMT
DIH itself may not be consuming so much memory. It also includes the
memory used by Solr.

Do you have a hard limit on 400MB  , is it not possible to increase it?

On Tue, Apr 14, 2009 at 11:09 AM, Mani Kumar <manikumarchauhan@gmail.com> wrote:
> Hi ILAN:
>
> Only one query is required to generate a document ...
> Here is my data-config.xml
>
> <dataConfig>
>    <dataSource type="JdbcDataSource" name="sp"
> driver="com.mysql.jdbc.Driver" url="jdbc:mysql://localhost/mydb_development"
> user="root" password="******" />
>    <document name="items">
>        <entity name="item" dataSource="sp" query="select * from items">
>            <field column="id" name="id" />
>            <field column="title" name="title" />
>        </entity>
>    </document>
> </dataConfig>
>
> and other useful info:
>
> mysql> select * from items
> +----------+
> | count(*) |
> +----------+
> |   900051 |
> +----------+
> 1 row in set (0.00 sec)
>
> Each record consist of id and title.
>
> id  is of type int(11) and title's avg. length is 50 chars.
>
>
> I am using tomcat with solr.
> here is the command i am using to start it
>
> ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
>
>
> Thanks! for help. I appreciate it.
>
> -Mani Kumar
>
> On Tue, Apr 14, 2009 at 2:31 AM, Ilan Rabinovitch <ilan@fonz.net> wrote:
>
>> Depending on your dataset and how your queries look you may very likely
>> need to increase to a larger heap size.  How many queries and rows are
>> required for each of your documents to be generated?
>>
>> Ilan
>>
>>
>> On 4/13/09 12:21 PM, Mani Kumar wrote:
>>
>>> Hi Shalin:
>>>
>>> Thanks for quick response!
>>>
>>> By defaults it was set to 1.93 MB.
>>> But i also tried it with following command:
>>>
>>> $  ./apache-tomcat-6.0.18/bin/startup.sh -Xmn50M -Xms300M -Xmx400M
>>>
>>> I also tried tricks given on
>>> http://wiki.apache.org/solr/DataImportHandlerFaq page.
>>>
>>> what should i try next ?
>>>
>>> Thanks!
>>> Mani Kumar
>>>
>>> On Tue, Apr 14, 2009 at 12:12 AM, Shalin Shekhar Mangar<
>>> shalinmangar@gmail.com>  wrote:
>>>
>>>  On Mon, Apr 13, 2009 at 11:57 PM, Mani Kumar<manikumarchauhan@gmail.com
>>>>
>>>>> wrote:
>>>>>
>>>>
>>>>  Hi All,
>>>>> I am trying to setup a Solr instance on my macbook.
>>>>>
>>>>> I get following errors when m trying to do a full db import ... please
>>>>>
>>>> help
>>>>
>>>>> me on this
>>>>>
>>>>> java.lang.OutOfMemoryError: Java heap space
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:400)
>>>>
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:221)
>>>>
>>>>>        at
>>>>>
>>>>> org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:164)
>>>>
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:312)
>>>>
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:370)
>>>>
>>>>>        at
>>>>>
>>>>>
>>>>> org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:351)
>>>>
>>>>> Caused by: java.lang.OutOfMemoryError: Java heap space
>>>>>        at com.mysql.jdbc.Buffer.<init>(Buffer.java:58)
>>>>>        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1444)
>>>>>        at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2840)
>>>>>
>>>>>
>>>>>  How much heap size have you allocated to the jvm?
>>>>
>>>> Also see http://wiki.apache.org/solr/DataImportHandlerFaq
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>>
>>>
>>
>> --
>> Ilan Rabinovitch
>> ilan@fonz.net
>>
>> ---
>> SCALE 7x: 2009 Southern California Linux Expo
>> Los Angeles, CA
>> http://www.socallinuxexpo.org
>>
>>
>



-- 
--Noble Paul

Mime
View raw message