Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of buschmic@gmail.com designates
 74.125.78.27 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:user-agent:mime-version:to:subject:references
         :in-reply-to:content-type:content-transfer-encoding;
        b=HpiWU49j2psyUnEkB9CMsg6XlO2NmsHIxTGiOVnLgxCmX5SqB+AlwFN6ATHAIcJdl/
         D/XLDJAz5+KI95OaMoeLZafcdQM1ZUTurpRox67G9pzr6xRjy6dXpmlsvUeAqbc+TaSb
         wmwItHSnuDZV2iWYkx5VMtgGcPSiJrE7S+tq0=
Message-ID: <4AF9AE6D.80507@gmail.com>
Date: Tue, 10 Nov 2009 10:18:21 -0800
From: Michael Busch <buschmic@gmail.com>
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US;
 rv:1.9.1.4pre) Gecko/20090915 Thunderbird/3.0b4
MIME-Version: 1.0
To: java-dev@lucene.apache.org
Subject: Re: Questions about doc store files (.cfx)
References: <4AF7D019.9020406@gmail.com>
	 <9ac0c6aa0911090256j2fc60ea9k53ac6fe939990ac5@mail.gmail.com>
	 <4AF830EA.30503@gmail.com>
	 <9ac0c6aa0911090900l4436974ye9e85690cf50d652@mail.gmail.com>
	 <4AF8C47B.1020100@gmail.com> <4AF8F4CC.6070609@gmail.com>
 <9ac0c6aa0911100157i7937b667o3d3baeb769ea7674@mail.gmail.com>
In-Reply-To: <9ac0c6aa0911100157i7937b667o3d3baeb769ea7674@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

On 11/10/09 1:57 AM, Michael McCandless wrote:
>
>> I think this is exactly what happens? I wrote a small test program that
>> creates a situation like mentioned above in the "expungeDelete" scenario. It
>> ends up with a docstore containing docs from two segments, but after
>> expungeDeletes only one segment references the docstore. The non-deleted
>> docs from the other segment end up in a new segment, so they are twice on
>> disk (once orphaned in the old docstore, once in the new segment).
>> Is that the desired behavior?
>>      
> Right this is what happens -- since segment C wasn't merged, it
> remains as the only segment still referencing the shared doc stores,
> and, yes, this does result in duplicate storage for some docs (until C
> is merged away).  IFD keeps track of whether a given set of doc stores
> is still referenced.
>
>    

OK, thanks for clarifying!

> I think in practice this should not result in too much duplication.
> If C is large, it's likely to have accumulated deletes as well.  If C
> is small, it's likely to get merged away in the course of normal
> merging.
>
>    

I agree - it shouldn't happen very often. I was just not sure how the 
current behavior in this corner case was and wanted to understand it.

> But, if we are really concerned with it, we could modify the merge
> policy to bias its selection on this ("remove stores that are wasting
> too much space") basis.
>    

I'm not too concerned, because I also don't think this should happen 
very often.

> I think this makes the parallel index job's simpler, right?  Ie, how
> the segments are sharing the stores within their own index does not
> restrict what merging is done.
>
>    

Yes exactly. It won't prevent us from keeping the parallel indexes 
independent in this regard.

Then the compound (.cfx and .cfs) files are rather orthogonal to this. I 
talked to Marvin on ApacheCon; in Lucy he wants to have all the compound 
file support in the store package, separately from the indexer. I think 
that would make sense in Lucene too, there's not really the need to have 
it tightly integrated in the IndexWriter and SegmentMerger. We can 
generalize the compound file concept further, so that with parallel 
indexes the files can be selected in either direction for inclusion in a 
compound file.

E.g. if we separated the inverted index and store, so that they are 
logically two parallel index components, then the .cfx file as it works 
now would contain files from two parallel index components (term vectors 
from inverted index, stored fields from the store). This is fine if you 
don't want to update those components individually and can remain this 
way for the default IndexWriter implementation. But if we generalize the 
compound concept, then people can alter this behavior to better suit 
their update requirements.

I think this would actually be a very clean design (even though it might 
sound complicated here).

> Mike
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>
>    


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org