lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Gründler <rob...@dubture.com>
Subject Re: Dataimport destroys our harddisks
Date Thu, 02 Dec 2010 14:59:38 GMT
On Dec 2, 2010, at 15:43 , Sven Almgren wrote:

> What Raid controller do you use, and what kernel version? (Assuming
> Linux). We hade problems during high load with a 3Ware raid controller
> and the current kernel for Ubuntu 10.04, we hade to downgrade the
> kernel...
> 
> The problem was a bug in the driver that only showed up with very high
> disk load (as is the case when doing imports)
> 

We're running freebsd:

RaidController  3ware 9500S-8
Corrupt unit: Raid-10 3725.27GB 256K Stripe Size without BBU
Freebsd 7.2, UFS Filesystem.



> /Sven
> 
> 2010/12/2 Robert Gründler <robert@dubture.com>:
>>> The very first thing I'd ask is "how much free space is on your disk
>>> when this occurs?" Is it possible that you're simply filling up your
>>> disk?
>> 
>> no, i've checked that already. all disks have plenty of space (they have
>> a capacity of 2TB, and are currently filled up to 20%.
>> 
>>> 
>>> do note that an optimize may require up to 2X the size of your index
>>> if/when it occurs. Are you sure you aren't optimizing as you add
>>> items to your index?
>>> 
>> 
>> index size is not a problem in our case. Our index currently has about 3GB.
>> 
>> What do you mean with "optimizing as you add items to your index"?
>> 
>>> But I've never heard of Solr causing hard disk crashes,
>> 
>> neither did we, and google is the same opinion.
>> 
>> One thing that i've found is the mergeFactor value:
>> 
>> http://wiki.apache.org/solr/SolrPerformanceFactors#mergeFactor
>> 
>> Our sysadmin speculates that maybe the chunk size of our raid/harddisks
>> and the segment size of the lucene index does not play well together.
>> 
>> Does the lucene segment size affect how the data is written to the disk?
>> 
>> 
>> thanks for your help.
>> 
>> 
>> -robert
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>> 
>>> Best
>>> Erick
>>> 
>>> 2010/12/2 Robert Gründler <robert@dubture.com>
>>> 
>>>> Hi,
>>>> 
>>>> we have a serious harddisk problem, and it's definitely related to a
>>>> full-import from a relational
>>>> database into a solr index.
>>>> 
>>>> The first time it happened on our development server, where the
>>>> raidcontroller crashed during a full-import
>>>> of ~ 8 Million documents. This happened 2 weeks ago, and in this period 2
>>>> of the harddisks where the solr
>>>> index files are located stopped working (we needed to replace them).
>>>> 
>>>> After the crash of the raid controller, we decided to move the development
>>>> of solr/index related stuff to our
>>>> local development machines.
>>>> 
>>>> Yesterday i was running another full-import of ~10 Million documents on my
>>>> local development machine,
>>>> and during the import, a harddisk failure occurred. Since this failure, my
>>>> harddisk activity seems to
>>>> be around 100% all the time, even if no solr server is running at all.
>>>> 
>>>> I've been googling the last 2 days to find some info about solr related
>>>> harddisk problems, but i didn't find anything
>>>> useful.
>>>> 
>>>> Are there any steps we need to take care of in respect to harddisk failures
>>>> when doing a full-import? Right now,
>>>> our steps look like this:
>>>> 
>>>> 1. Delete the current index
>>>> 2. Restart solr, to load the updated schemas
>>>> 3. Start the full import
>>>> 
>>>> Initially, the solr index and the relational database were located on the
>>>> same harddisk. After the crash, we moved
>>>> the index to a separate harddisk, but nevertheless this harddisk crashed
>>>> too.
>>>> 
>>>> I'd really appreciate any hints on what we might do wrong when importing
>>>> data, as we can't release this
>>>> on our production servers when there's the risk of harddisk failures.
>>>> 
>>>> 
>>>> thanks.
>>>> 
>>>> 
>>>> -robert
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>> 
>> 


Mime
View raw message