maven-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cody Wayne Holmes (Jira)" <j...@apache.org>
Subject [jira] [Commented] (MNG-6836) Too Many Open Files for large projects
Date Thu, 02 Jan 2020 16:15:00 GMT

    [ https://issues.apache.org/jira/browse/MNG-6836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17006898#comment-17006898
] 

Cody Wayne Holmes commented on MNG-6836:
----------------------------------------

Sure, some of the numbers for one project are: (Note: these are approximate numbers, but
are fairly close to the real world example)

 

Number of modules: 50 (3 are packaging type pom)

Relative total build time using 9 build threads: 22 mins

Total Project dependencies: 918

Total number of plugins used: 31

Size of fresh local m2 after a build" ~1GB (4 - 6k files)

Ulimit set when number of files exceeded: 1024s/4096h (It is hitting the hard limit of 4096)

> Too Many Open Files for large projects
> --------------------------------------
>
>                 Key: MNG-6836
>                 URL: https://issues.apache.org/jira/browse/MNG-6836
>             Project: Maven
>          Issue Type: Bug
>          Components: Plugins and Lifecycle
>    Affects Versions: 3.5.2, 3.5.3, 3.5.4, 3.6.0, 3.6.1, 3.6.2, 3.6.3
>            Reporter: Cody Wayne Holmes
>            Priority: Major
>
> This issue seems to be that all plugins classloaders are created and then cached, and
plugins are internally creating classloaders that are not being closed and therefore leaving
open file handles to jar files waiting to be finalized.
>  
> When a project gets big enough with enough modules and plugins than this becomes a problem
as the project will reach a large number of open jar files by the time it completes and if
the file system is not set up to allow the standard ulimit to be exceeded, then this will
cause build failures.
>  
> One plugin example that opens and does not close a URLClassloader is the [maven-pmd-plugin|#L796]].
As you can see it calls the method to prepend the classpath for the plugin to run. This method
then [creates|#L196]] a URLClassloader that is not closed and must be finalized.
>  
> With the deprecation of the finalizer and the knowledge that it is not a good mechanism
for closing system resources reliably, this also causes issues for larger projects with a
lot of modules and a slightly larger build time.
>  
> I am not sure what can be done about this issue in the maven lifecycle, but just wanted
to ask the question and find out if anyone else has seen similar issues or knows of a way
to resolve this open file handles problem. And possibly find a way to limit the number of
open file handles to the downloaded artifacts.
>  
> Thanks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message