lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From tushar kapoor <tushar_kapoor...@rediffmail.com>
Subject Re: Create incremental snapshot
Date Sun, 12 Jul 2009 10:52:25 GMT

Thanks for the reply Asif. We have already tried removing the optimization
step. Unfortunately the commit command alone is also causing an identical
behaviour . Is there any thing else that we are missing ?


Asif Rahman wrote:
> 
> Tushar:
> 
> Is it necessary to do the optimize on each iteration?  When you run an
> optimize, the entire index is rewritten.  Thus each index file can have at
> most one hard link and each snapshot will consume the full amount of space
> on your disk.
> 
> Asir
> 
> On Thu, Jul 9, 2009 at 3:26 AM, tushar kapoor <
> tushar_kapoor_47@rediffmail.com> wrote:
> 
>>
>> What I gather from this discussion is -
>>
>> 1. Snapshots are always hard links and not actual files so they cannot
>> possibly consume the same amount    of space.
>> 2. Snapshots contain hard links to existing docs + delta docs.
>>
>> We are facing a situation wherein the snapshot occupies the same space as
>> the actual indexes thus violating the first point.
>> We have a batch processing scheme for refreshing indexes. the steps we
>> follow are -
>>
>> 1. Delete 200 documents in one go.
>> 2. Do an optimize.
>> 3. Create the 200 documents deleted earlier.
>> 4. Do a commit.
>>
>> This process continues for around 160,000 documents i.e. 800 times and by
>> the end of it we have 800 snapshots.
>>
>> The size of actual indexes is 200 Mb and remarkably all the 800 snapshots
>> are of size around 200 Mb each. In effect this process consumes around
>> 160
>> Gb space on our disks. This is causing a lot of pain right now.
>>
>> My concern are - Is our understanding of the snapshooter correct ? Should
>> this massive space consumption be happening at all ? Are we missing
>> something critical ?
>>
>> Regards,
>> Tushar.
>>
>> Shalin Shekhar Mangar wrote:
>> >
>> > On Sat, Apr 18, 2009 at 1:06 PM, Koushik Mitra
>> > <Koushik_Mitra@infosys.com>wrote:
>> >
>> >> Ok....
>> >>
>> >> If these are hard links, then where does the index data get stored?
>> Those
>> >> must be getting stored somewhere in the file system.
>> >>
>> >
>> > Yes, of course they are stored on disk. The hard links are created from
>> > the
>> > actual files inside the index directory. When those older files are
>> > deleted
>> > by Solr, they are still left on the disk if at least one hard link to
>> that
>> > file exists. If you are looking for how to clean old snapshots, you
>> could
>> > use the snapcleaner script.
>> >
>> > Is that what you wanted to do?
>> >
>> > --
>> > Regards,
>> > Shalin Shekhar Mangar.
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/Create-incremental-snapshot-tp23109877p24405434.html
>> Sent from the Solr - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> -- 
> Asif Rahman
> Lead Engineer - NewsCred
> asif@newscred.com
> http://platform.newscred.com
> 
> 
:-((
-- 
View this message in context: http://www.nabble.com/Create-incremental-snapshot-tp23109877p24447593.html
Sent from the Solr - User mailing list archive at Nabble.com.


Mime
View raw message