lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: WriteLineDocTask does not release resources
Date Sat, 11 Apr 2009 09:59:57 GMT
Actually, in LUCENE-1516 I added empty PerfTask.close() for exactly
this purpose.  (Probably you need to svn up).

(The NearRealtimeReaderTask needed to close its reader).

Why do we need a separate Map to track separate classes that close
resources?  Why not simply ask each task to implement close() if it
needs to?  (The aggregate task TaskSequence just forward the request
to its children).

For things like closing files, compressors, etc, I don't think we
should create an explicit CloseResourcesTask -- I think the framework
should simply close such resources at the right time.

For "larger" resources (my IndexReader, IndexWriter) I do think we
should make it explicit when they get closed?


On Sat, Apr 11, 2009 at 2:09 AM, Shai Erera <> wrote:
> In fact, the more I think of it, I think it can be generalized even further.
> Create a TestResource abstract class with an abstract method release(). Then
> allow to aggregate TestResources in PerfRunData, either in a map (Task,
> TestResource) or just as a list. We can then either: (1) create a
> ReleaseTestResourcesTask, which iterates through the list/map and releases
> all, or (2) do this at the end of the test run or (3) do both.
> This can replace the CloseIndex and CloseReader tasks, although not delete
> the classes as someone might wanna call them in the middle of a test.
> This approach can also cover LineDocMaker and EnwikiDocMaker (and I'm sure
> others as well) by allowing to close their input streams (which are not
> closed today).
> I think a map of Task, TestResource is better since we can also add a task
> ClearTestResource which accepts a task name as a parameter.
> How's that sound? I can open issue if that sounds acceptable.
> On Sat, Apr 11, 2009 at 8:04 AM, Shai Erera <> wrote:
>> WriteLineDocTask instantiates a BufferedWriter, but never closes it. This
>> causes some problems in LUCENE-1591 since I want to wrap CBZip2OutputStream,
>> and the stream has to be closed in order for the archive to be valid
>> (flush() is not enough).
>> Unlike DocMaker, which has a resetInputs method, tasks have no such
>> *finalizer* method which can be called upon test completion, or even by
>> ResetInputsTask. I assume that's because tasks are viewed as stateless
>> objects, or at least very lightweight. Indeed, all tasks (besides this one)
>> use the information stored in PerfRunData to operate, and thus are
>> "stateless". Someone can include a *finalizer* task, such as CloseIndex, if
>> the test used any resource that need to be closed.
>> setup() and tearDown() are called once for every doLogic(), so we cannot
>> release such resources there.
>> I was thinking how to solve it:
>> * Include a map of <Task, TaskResources> in PerfRunData, and create a new
>> task WriteLineDocFinalizerTask which will get the resources used by
>> WriteLineDocTask and close them (in this case the Writer used to output
>> data). WLDT will store the writer in this map.
>> * Introduce PerfTask.releaseResources() which will have empty impl in
>> PerfTask, and extended in WLDT to close the writer. That method will be
>> called upon test completion (on all tasks).
>> I like option (1) better. TaskResources can be just an interface, with no
>> methods, and each task will have its own implementation.
>> What do you think?
>> Shai

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message