lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <gsing...@apache.org>
Subject Re: Deduplication patch not working in nightly build
Date Sat, 10 Jan 2009 15:24:35 GMT
I've seen similar errors when large background merges happen while  
looping in a result set.  See http://lucene.grantingersoll.com/2008/07/16/mysql-solr-and-communications-link-failure/



On Jan 9, 2009, at 12:50 PM, Mark Miller wrote:

> Your basically writing segments more often now, and somehow avoiding  
> a longer merge I think. Also, likely, deduplication is probably  
> adding enough extra data to your index to hit a sweet spot where a  
> merge is too long. Or something to that effect - MySql is especially  
> sensitive to timeouts when doing a select * on a huge db in my  
> testing. I didnt understand your answer on the autocommit - I take  
> it you are using it? Or no?
>
> All a guess, but it def points to a merge taking a bit long and  
> causing a timeout. I think you can relax the MySql timeout settings  
> if that is it.
>
> I'd like to get to the bottom of this as well, so any other info you  
> can provide would be great.
>
> - Mark
>
> Marc Sturlese wrote:
>> Hey Shalin,
>>
>> In the begining (when the error was appearing) i had  
>> <ramBufferSizeMB>32</ramBufferSizeMB>
>> and no maxBufferedDocs set
>>
>> Now I have:
>> <ramBufferSizeMB>32</ramBufferSizeMB>
>> <maxBufferedDocs>50</maxBufferedDocs>
>>
>> I think taht setting maxBufferedDocs to 50 I am forcing more disk  
>> writting
>> than I would like... but at least it works fine (but a bit  
>> slower,opiously).
>>
>> I keep saying that the most weird thing is that I don't have that  
>> problem
>> using solr1.3, just with the nightly...
>>
>> Even that it's good that it works well now, would be great if  
>> someone can
>> give me an explanation why this is happening
>>
>>
>> Shalin Shekhar Mangar wrote:
>>
>>> On Fri, Jan 9, 2009 at 9:23 PM, Marc Sturlese
>>> <marc.sturlese@gmail.com>wrote:
>>>
>>>
>>>> hey there,
>>>> I hadn't autoCommit set to true but I have it sorted!!!!! The error
>>>> stopped
>>>> appearing after setting the property maxBufferedDocs in  
>>>> solrconfig.xml. I
>>>> can't exactly undersand why but it just worked.
>>>> Anyway, maxBufferedDocs is deprecaded, would ramBufferSizeMB do  
>>>> the same?
>>>>
>>>>
>>>>
>>> What I find strange is this line in the exception:
>>> "Last packet sent to the server was 202481 ms ago."
>>>
>>> Something took very very long to complete and the connection got  
>>> closed by
>>> the time the next row was fetched from the opened resultset.
>>>
>>> Just curious, what was the previous value of maxBufferedDocs and  
>>> what did
>>> you change it to?
>>>
>>>
>>>
>>>> --
>>>> View this message in context:
>>>> http://www.nabble.com/Deduplication-patch-not-working-in-nightly-build-tp21287327p21374908.html
>>>> Sent from the Solr - User mailing list archive at Nabble.com.
>>>>
>>>>
>>>>
>>> -- 
>>> Regards,
>>> Shalin Shekhar Mangar.
>>>
>>>
>>>
>>
>>
>

--------------------------
Grant Ingersoll

Lucene Helpful Hints:
http://wiki.apache.org/lucene-java/BasicsOfPerformance
http://wiki.apache.org/lucene-java/LuceneFAQ











Mime
View raw message