Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAOL5jnX9mBngzQzp=PwQs4jsPFrsCH=RT45=QW9CrzwJ=p4yYA@mail.gmail.com>
References: 
 <CAOL5jnX9mBngzQzp=PwQs4jsPFrsCH=RT45=QW9CrzwJ=p4yYA@mail.gmail.com>
From: Michael McCandless <lucene@mikemccandless.com>
Date: Wed, 6 Jun 2012 07:37:21 -0400
Message-ID: 
 <CAL8PwkZb6xQNu3GDeFWd90rAOfw3P4kJM_nrNOWQkb-BgrAcjg@mail.gmail.com>
Subject: Re: IndexCommit.delete() outside of IndexDeletionPolicy
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

I think this use case makes sense; such logic (for a distributed / ref
counted deletion policy) would make a nice contribution ... it's the
"proper" way to delete commits when multiple nodes are in use (vs eg
using a timeout deletion policy).

You can actually do it today: call IndexWriter.deleteUnusedFiles.
That visits the deletion policy and then you have a chance to delete
commit points (it'd mean you have to set a real deletion policy on the
writer, which in turn goes and checks the reference counts across all
nodes).

Mike McCandless

http://blog.mikemccandless.com


On Wed, Jun 6, 2012 at 7:16 AM, Colin Goodheart-Smithe
<colings86.dev@googlemail.com> wrote:
> I was looking at the Lucene API for IndexCommit and noticed that the
> JavaDoc states that
>
> *'Decision that a commit-point should be deleted is taken by the
> IndexDeletionPolicy<http://lucene.apache.org/core/3_6_0/api/all/org/apach=
e/lucene/index/IndexDeletionPolicy.html>
> in
> effect and therefore this should only be called by its
> onInit()<http://lucene.apache.org/core/3_6_0/api/all/org/apache/lucene/in=
dex/IndexDeletionPolicy.html#onInit(java.util.List)>
> =A0or onCommit()<http://lucene.apache.org/core/3_6_0/api/all/org/apache/l=
ucene/index/IndexDeletionPolicy.html#onCommit(java.util.List)>
> =A0methods.'*
> (
> http://lucene.apache.org/core/3_6_0/api/all/org/apache/lucene/index/Index=
Commit.html#delete()
> =A0)
>
> I was wondering why this is the case and whether deleting IndexCommits
> outside of a IndexDeletionPolicy is actually a bad idea?
>
> To put some context around this I am looking to implement a deletion poli=
cy
> which is independant of the IndexWriter commit and more dependant on
> Processes using particular Commit points being finished with it.
> The logic would look something like the following and state would be stor=
ed
> in something like ZooKeeper so I can have use of ephremal nodes and watch=
er
> events:
>
> =A0 - IndexWriters would have a NoDeletionPolicy set
> =A0 - Each time a process opens a session it registers an ephremal node
> =A0 - The session is assigned the current (latest) commit point
> =A0 - Each time a process removes the node (either through crashing or
> =A0 having finished the job) a watch event is fired where a separate proc=
ess
> =A0 will delete the commit point the process was using if no other proces=
ses
> =A0 are using the commit point and if it is not the latest commit point
>
> Processes may have fairly long running sessions so across all the process=
es
> a reasonable number of commit points might be in use. =A0I don't really w=
ant
> to have to wait for a commit from the IndexWriter (which may not happen f=
or
> a while) to clear up the older commit points I no longer need. =A0Would t=
his
> logic pose any issues given that it is going to be deleting Commit points
> outside of the IndexDeletionPolicy

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org