commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jochen Wiedmann (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FILEUPLOAD-288) Linux dentry cache memory leak due to inappropriate temporary file detection
Date Thu, 23 Nov 2017 20:44:00 GMT

     [ https://issues.apache.org/jira/browse/FILEUPLOAD-288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jochen Wiedmann updated FILEUPLOAD-288:
---------------------------------------
    Fix Version/s: 1.4
                   Nightly Builds

> Linux dentry cache memory leak due to inappropriate temporary file detection
> ----------------------------------------------------------------------------
>
>                 Key: FILEUPLOAD-288
>                 URL: https://issues.apache.org/jira/browse/FILEUPLOAD-288
>             Project: Commons FileUpload
>          Issue Type: Improvement
>         Environment: OS version: CentOS release 6.7 (Final)
> Kenel: 2.6.32-573.26.1.el6.x86_64
> JVM: Oracle 1.8.0_102-b14
> Tomcat: 8.0.33.0
> commons-fileupload : 1.3.3
>            Reporter: fangwentong
>             Fix For: Nightly Builds, 1.4
>
>
> We use commons-fileupload:1.3.3 to handle multipart request, param {{file-size-threshold}}
was set to 200K. Our API handle multipart request in high QPS, the size of request body is
about 20KB, which means those parts was saved in memory, not written to disk temporary file.
> But, in our production environment,  size of dentry cache grows up in high speed. here
are some analysis:
> in {{strace}} result, plenty of {{stat}} system call invoked for tmp files which not
exists. 
> {noformat}
> [pid 111310] <... stat resumed> {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0
> [pid 57844] <... stat resumed> 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624923.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624922.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624921.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624920.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624919.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624918.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 69798] stat("/tmp",  <unfinished ...>
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624917.tmp",  <unfinished
...>
> [pid 69798] <... stat resumed> {st_mode=S_IFDIR|0755, st_size=6, ...}) = 0
> [pid 57844] <... stat resumed> 0x7f08e80b84b0) = -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624916.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624915.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624914.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624913.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624912.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624911.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624910.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624909.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624908.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624907.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624906.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624905.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624904.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624903.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624902.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624901.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624900.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624899.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624898.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624897.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624896.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624895.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624894.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624893.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624892.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624891.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624890.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624889.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624888.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624887.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624886.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624885.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 57844] stat("/tmp/upload_75a3e205_c3f1_43e7_8165_34f6a7a02227_518624884.tmp", 0x7f08e80b84b0)
= -1 ENOENT (No such file or directory)
> [pid 78454] --- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=57810, si_uid=605}
---{noformat}
> in {{slabtop}} result , dentry cache taken up 22GB+ memory.
> {noformat}
>  Active / Total Objects (% used)    : 233715492 / 234153507 (99.8%)
>  Active / Total Slabs (% used)      : 7858579 / 7858606 (100.0%)
>  Active / Total Caches (% used)     : 108 / 200 (54.0%)
>  Active / Total Size (% used)       : 29322726.34K / 29380106.43K (99.8%)
>  Minimum / Average / Maximum Object : 0.02K / 0.12K / 4096.00K
>   OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME
> 116095303 116095295  26%    0.06K 1967717       59   7870868K size-64
> 116090260 116090260  26%    0.19K 5804513       20  23218052K dentry
> {noformat}
> finally, I think the root cause is:in temporary file cleanup policy defined at {{DiskFileItem#finalize}},
call {{outputFile.exists()}} for every File, even if the part not write to disk.
> {code:java}
>     @Override
>     protected void finalize() {
>         if (dfos == null) {
>             return;
>         }
>         File outputFile = dfos.getFile();
>         if (outputFile != null && outputFile.exists()) {
>             outputFile.delete();
>         }
>     }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message