commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefan Bodewig (JIRA)" <>
Subject [jira] [Commented] (COMPRESS-446) Resource Leak in ParallelScatterZipCreator#writeTo(ZipArchiveOutputStream)
Date Sat, 31 Mar 2018 09:26:00 GMT


Stefan Bodewig commented on COMPRESS-446:

Hmm, I don't see this as serious as you seem to do.

The resource leak happens if one of the parallel threads throws an exception which will likely
propagate to the caller and in code that hasn't been crafted  as carefully as Christoph's
will kill the whole process rendering the resource leak moot. This is an edge case in a class
that isn't likely to be used by many people at all.

Do you consider this more serious than the "usual" bugs?

> Resource Leak in ParallelScatterZipCreator#writeTo(ZipArchiveOutputStream)
> --------------------------------------------------------------------------
>                 Key: COMPRESS-446
>                 URL:
>             Project: Commons Compress
>          Issue Type: Bug
>          Components: Archivers
>    Affects Versions: 1.16.1
>         Environment: The application was running inside a Docker container, the JVM had
about 1.7 GByte heap space.
>            Reporter: Christoph Ludwig
>            Priority: Major
>             Fix For: 1.17
> Before it does anything else, {{ParallelScatterZipCreator#writeTo(ZipArchiveOutputStream)}}
loops over all futures returned by the creator`s executor service and calls {{Future#get()}}.
This will block until the future's computation is completed, respectively - i.e., until all
entries have been written to the thread-local scatter streams.
> However, if the computation of a future fails, then {{Future#get()}} can also throw an
exception. This exception escapes {{ParallelScatterZipCreator#writeTo(ZipArchiveOutputStream)}}
before the executor service is shut down. The latter means that also the thread-local variables
in the executor service's threads and all objects referenced by them continue to exist and
cannot be reclaimed by the GC.
> I encountered this situation when - while processing an archive with 130,000 documents
- the JVM threw an {{OutOfMemoryError}}. The application was not able to recover from this
OOM error because most of the heap was occupied by objects reachable from the executor service's
> Of course, the OOM is mostly the fault of my own code; I will be able to work around
the "leaked" executor service because I supply it in the first place and can therefore shut
it down if I detect an error situation.  
> The effect would be the same, though, if, say, {{Future#get()}} throws an {{InterruptedException}}.
Therefore, {{ParallelScatterZipCreator#writeTo(ZipArchiveOutputStream)}}  should either shut
down and release all resources if it cannot complete its task due to an Exception thrown by
a future or it should offer a reasonable recovery strategy. 

This message was sent by Atlassian JIRA

View raw message