lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] [Commented] (LUCENE-4190) IndexWriter deletes non-Lucene files
Date Wed, 04 Jul 2012 13:42:34 GMT


Michael McCandless commented on LUCENE-4190:

bq. Many of the lucene+facet deployments that I know of store the taxonomy index as a sub-directory
of the search index

We won't delete directories, just files.

bq. Also, we've been storing other files in the index directory too ... this new feature will
affect such existing deployments.

Yeah ... better to move them elsewhere or to a sub dir?

bq.  I can assume it's related to IW not knowing which files to delete when a segment is no
longer needed, because Codecs can pick their own file names

Right: it's easy to track the positive set (files referenced by current segments), what's
harder is the negative set (files created in the past but no longer referenced).

bq. If we had an instance which kept track of all files that were created, e.g. every Codec
would register the files there (if it wants to protect from their deletion), would make the
decision of which files to delete easier?

In theory it would ... but this would add a fair amount of complexity (we'd have to save this
list of files into segments_N).  In fact long ago Lucene did this (it had a deletable file
which stored the list of files previously created and now to-be-deleted).
> IndexWriter deletes non-Lucene files
> ------------------------------------
>                 Key: LUCENE-4190
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>            Reporter: Michael McCandless
>            Assignee: Robert Muir
>             Fix For: 4.0
>         Attachments: LUCENE-4190.patch
> Carl Austin raised a good issue in a comment on my Lucene 4.0.0 alpha blog post:
> IndexWriter will now (as of 4.0) delete all foreign files from the index directory. 
We made this change because Codecs are free to write to any files now, so the space of filenames
is hard to "bound".
> But if the user accidentally uses the wrong directory (eg c:/) then we will in fact delete
important stuff.
> I think we can at least use some simple criteria (must start with _, maybe must fit certain
pattern eg _<base36>(_X).Y), so we are much less likely to delete a non-Lucene file....

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message