lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruno Mannina <bmann...@free.fr>
Subject Re: Indexation Speed?
Date Wed, 20 Jun 2012 08:40:53 GMT
Ok thanks for this information,

Le 20/06/2012 05:44, Lance Norskog a écrit :
> M. Della Bitta is right- we're not talking about post.jar, but starting Solr:
>
> java -xMx300m -jar start.jar
>
> On Tue, Jun 19, 2012 at 10:05 AM, Erick Erickson
> <erickerickson@gmail.com>  wrote:
>> Well, it _used_ to be defaulted in the code, but on looking at 3.6 it's seems
>> like it defaults to Integer.MAX_VALUE, so you're fine....
>>
>> And it's all deprecated in 4.x, will be gone
>>
>> Best
>> Erick
>>
>> On Tue, Jun 19, 2012 at 7:07 AM, Bruno Mannina<bmannina@free.fr>  wrote:
>>> Actually -Xmx512m and no effect
>>>
>>> Concerning  maxFieldLength, no problem it's commented
>>>
>>> Le 19/06/2012 13:02, Erick Erickson a écrit :
>>>
>>>> Then try -Xmx600M
>>>> next try -Xmx900M
>>>>
>>>>
>>>> etc. The idea is to bump things on separate runs.
>>>>
>>>> But be a little cautious here. Look in your solrconfig.xml file, you'll
>>>> see
>>>> a commented-out line
>>>> <maxFieldLength>10000</maxFieldLength>
>>>>
>>>> The default behavior for Solr/Lucene is to index the first 10,000 tokens
>>>> (not characters, think of tokens as words for not) in each
>>>> document and throw the rest on the floor. At the sizes you're talking
>>>> about,
>>>> that's probably not a problem, but do be aware of it.
>>>>
>>>> Best
>>>> Erick
>>>>
>>>> On Tue, Jun 19, 2012 at 5:44 AM, Bruno Mannina<bmannina@free.fr>  
 wrote:
>>>>> Like that?
>>>>>
>>>>> java -Xmx300m -jar post.jar myfile.xml
>>>>>
>>>>>
>>>>>
>>>>> Le 19/06/2012 11:11, Lance Norskog a écrit :
>>>>>
>>>>>> Ah! Java memory size is a java command line option:
>>>>>>
>>>>>>
>>>>>> http://javahowto.blogspot.com/2006/06/6-common-errors-in-setting-java-heap.html
>>>>>>
>>>>>> You would try increasing the memory size in stages up to maybe 300m.
>>>>>>
>>>>>> On Tue, Jun 19, 2012 at 2:04 AM, Bruno Mannina<bmannina@free.fr>
>>>>>>   wrote:
>>>>>>>
>>>>>>> Le 19/06/2012 10:51, Lance Norskog a écrit :
>>>>>>>
>>>>>>>> 675 doc/s is respectable for that server. You might move
the memory
>>>>>>>> allocated to Java up and down- there is a balance between
amount of
>>>>>>>> memory in Java v.s. the OS disk buffer.
>>>>>>>
>>>>>>> How can I do that ? is there an option during my command line
or in a
>>>>>>> config
>>>>>>> file?
>>>>>>> sorry for this newbie question :(
>>>>>>>
>>>>>>>
>>>>>>>> And, of course, use the latest trunk.
>>>>>>> Solr 3.6
>>>>>>>
>>>>>>>
>>>>>>>> On Tue, Jun 19, 2012 at 12:10 AM, Bruno Mannina<bmannina@free.fr>
>>>>>>>>   wrote:
>>>>>>>>> Correction: file size is 40 Mo !!!
>>>>>>>>>
>>>>>>>>> Le 19/06/2012 09:09, Bruno Mannina a écrit :
>>>>>>>>>
>>>>>>>>>> Dear All,
>>>>>>>>>>
>>>>>>>>>> I would like to know if the indexation speed is right.
>>>>>>>>>>
>>>>>>>>>> I have a 40Go file size with around 27 000 docs inside.
>>>>>>>>>> I index around 20 fields,
>>>>>>>>>>
>>>>>>>>>> My (old) test server is a DualCore 3.06GHz Intel
Xeon with only 1Go
>>>>>>>>>> Ram
>>>>>>>>>>
>>>>>>>>>> The file takes 40 seconds with the command line:
>>>>>>>>>> java -jar post.jar myfile.xml
>>>>>>>>>>
>>>>>>>>>> Could I increase this speed or reduce this time?
>>>>>>>>>>
>>>>>>>>>> Thanks a lot,
>>>>>>>>>> PS: Newbie user
>>>>>>>>>>
>>>>>>>>>>
>
>


Mime
View raw message