hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6628) Potential memory leak in CryptoOutputStream
Date Tue, 17 May 2016 01:01:13 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15285773#comment-15285773
] 

Hadoop QA commented on MAPREDUCE-6628:
--------------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 11s {color} | {color:blue}
Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green}
The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color}
| {color:green} The patch appears to include 1 new or modified test files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s {color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 8s {color} | {color:red}
root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 19s {color} |
{color:green} trunk passed with JDK v1.8.0_91 {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 9s {color} | {color:red}
root in trunk failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 56s {color}
| {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 16s {color} | {color:red}
hadoop-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 14s {color} | {color:red}
hadoop-mapreduce-client-core in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 14s {color} | {color:red}
hadoop-mapreduce-client-jobclient in trunk failed. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m 8s {color} | {color:red}
hadoop-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 8s {color} | {color:red}
hadoop-common in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 11s {color} | {color:red}
hadoop-mapreduce-client-core in trunk failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 11s {color} | {color:red}
hadoop-mapreduce-client-jobclient in trunk failed. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 7s {color} | {color:red}
hadoop-common in trunk failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 9s {color} | {color:red}
hadoop-common in trunk failed with JDK v1.7.0_95. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s {color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 8s {color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 12s {color} | {color:red}
hadoop-mapreduce-client-core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 11s {color} | {color:red}
hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s {color} |
{color:green} the patch passed with JDK v1.8.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 48s {color} | {color:green}
the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 10s {color} | {color:red}
root in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 10s {color} | {color:red}
root in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m 26s {color} | {color:red}
root: patch generated 32 new + 232 unchanged - 4 fixed = 264 total (was 236) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 12s {color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 14s {color} | {color:red}
hadoop-mapreduce-client-core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 13s {color} | {color:red}
hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvneclipse {color} | {color:red} 0m 8s {color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s {color} | {color:red}
The patch has 49 line(s) that end in whitespace. Use git apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 8s {color} | {color:red}
hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 12s {color} | {color:red}
hadoop-mapreduce-client-core in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 10s {color} | {color:red}
hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 29s {color} |
{color:green} hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91
with JDK v1.8.0_91 generated 0 new + 2497 unchanged - 12 fixed = 2497 total (was 2509) {color}
|
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 8s {color} | {color:red}
hadoop-common in the patch failed with JDK v1.8.0_91. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s {color} |
{color:green} hadoop-mapreduce-client-core in the patch passed with JDK v1.8.0_91. {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} |
{color:green} hadoop-mapreduce-client-jobclient in the patch passed with JDK v1.8.0_91. {color}
|
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 9s {color} | {color:red}
hadoop-common in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 7s {color} | {color:red}
hadoop-common in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 19s {color} | {color:red}
hadoop-mapreduce-client-core in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 106m 41s {color} | {color:red}
hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.8.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 15s {color} | {color:red}
hadoop-common in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 21s {color} | {color:red}
hadoop-mapreduce-client-core in the patch failed with JDK v1.7.0_95. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 22s {color} | {color:red}
hadoop-mapreduce-client-jobclient in the patch failed with JDK v1.7.0_95. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s {color}
| {color:green} Patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 134m 44s {color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_91 Failed junit tests | hadoop.mapred.TestMRCJCFileOutputCommitter |
|   | hadoop.mapred.TestMiniMRChildTask |
\\
\\
|| Subsystem || Report/Notes ||
| Docker |  Image:yetus/hadoop:cf2ee45 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12804278/MAPREDUCE-6628.006.patch
|
| JIRA Issue | MAPREDUCE-6628 |
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  findbugs
 checkstyle  |
| uname | Linux f5407abf5baf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12
UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4b55642 |
| Default Java | 1.7.0_95 |
| Multi-JDK versions |  /usr/lib/jvm/java-8-oracle:1.8.0_91 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_95
|
| mvninstall | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvninstall-root.txt
|
| compile | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-compile-root-jdk1.7.0_95.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvnsite-hadoop-common-project_hadoop-common.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| mvneclipse | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-mvneclipse-hadoop-common-project_hadoop-common.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-findbugs-hadoop-common-project_hadoop-common.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/branch-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
|
| mvninstall | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvninstall-hadoop-common-project_hadoop-common.txt
|
| mvninstall | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| mvninstall | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| compile | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-compile-root-jdk1.7.0_95.txt
|
| javac | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-compile-root-jdk1.7.0_95.txt
|
| checkstyle | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/diff-checkstyle-root.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvnsite-hadoop-common-project_hadoop-common.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| mvnsite | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| mvneclipse | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-mvneclipse-hadoop-common-project_hadoop-common.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/whitespace-eol.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-findbugs-hadoop-common-project_hadoop-common.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core.txt
|
| findbugs | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-findbugs-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
|
| javadoc | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-javadoc-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
|
| unit | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.8.0_91.txt
|
| unit | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.8.0_91.txt
|
| unit | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt
|
| unit | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-common-project_hadoop-common-jdk1.7.0_95.txt
|
| unit | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-core-jdk1.7.0_95.txt
|
| unit | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.7.0_95.txt
|
| unit test logs |  https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_91.txt
|
| JDK v1.7.0_95  Test Results | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/testReport/
|
| modules | C:  hadoop-common-project/hadoop-common   hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
  hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient  U: .
|
| Console output | https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6502/console |
| Powered by | Apache Yetus 0.2.0   http://yetus.apache.org |


This message was automatically generated.



> Potential memory leak in CryptoOutputStream
> -------------------------------------------
>
>                 Key: MAPREDUCE-6628
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6628
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: security
>    Affects Versions: 2.6.4
>            Reporter: Mariappan Asokan
>            Assignee: Mariappan Asokan
>         Attachments: MAPREDUCE-6628.001.patch, MAPREDUCE-6628.002.patch, MAPREDUCE-6628.003.patch,
MAPREDUCE-6628.004.patch, MAPREDUCE-6628.005.patch, MAPREDUCE-6628.006.patch
>
>
> There is a potential memory leak in {{CryptoOutputStream.java.}}  It allocates two direct
byte buffers ({{inBuffer}} and {{outBuffer}}) that get freed when {{close()}} method is called.
 Most of the time, {{close()}} method is called.  However, when writing to intermediate Map
output file or the spill files in {{MapTask}}, {{close()}} is never called since calling so
 would close the underlying stream which is not desirable.  There is a single underlying physical
stream that contains multiple logical streams one per partition of Map output.  
> By default the amount of memory allocated per byte buffer is 128 KB and  so the total
memory allocated is 256 KB,  This may not sound much.  However, if the number of partitions
(or number of reducers) is large (in the hundreds) and/or there are spill files created in
{{MapTask}}, this can grow into a few hundred MB. 
> I can think of two ways to address this issue:
> h2. Possible Fix - 1
> According to JDK documentation:
> {quote}
> The contents of direct buffers may reside outside of the normal garbage-collected heap,
and so their impact upon the memory footprint of an application might not be obvious.  It
is therefore recommended that direct buffers be allocated primarily for large, long-lived
buffers that are subject to the underlying system's native I/O operations.  In general it
is best to allocate direct buffers only when they yield a measureable gain in program performance.
> {quote}
> It is not clear to me whether there is any benefit of allocating direct byte buffers
in {{CryptoOutputStream.java}}.  In fact, there is a slight CPU overhead in moving data from
{{outBuffer}} to a temporary byte array as per the following code in {{CryptoOutputStream.java}}.
> {code}
>     /*
>      * If underlying stream supports {@link ByteBuffer} write in future, needs
>      * refine here. 
>      */
>     final byte[] tmp = getTmpBuf();
>     outBuffer.get(tmp, 0, len);
>     out.write(tmp, 0, len);
> {code}
> Even if the underlying stream supports direct byte buffer IO (or direct IO in OS parlance),
it is not clear whether it will yield any measurable performance gain.
> The fix would be to allocate a ByteBuffer on the heap for inBuffer and wrap a byte array
in a {{ByteBuffer}} for {{outBuffer}}.  By the way, the {{inBuffer}} and {{outBuffer}} have
to be {{ByteBuffer}} as demanded by the {{encrypt()}} method in {{Encryptor}}.
> h2. Possible Fix - 2
> Assuming that we want to keep the buffers as direct byte buffers, we can create a new
constructor to {{CryptoOutputStream}} and pass a boolean flag {{ownOutputStream}} to indicate
whether the underlying stream will be owned by {{CryptoOutputStream}}. If it is true, then
calling the {{close()}} method will close the underlying stream.  Otherwise, when {{close()}}
is called only the direct byte buffers will be freed and the underlying stream will not be
closed.
> The scope of changes for this fix will be somewhat wider.  We need to modify {{MapTask.java}},
{{CryptoUtils.java}}, and {{CryptoFSDataOutputStream.java}} as well to pass the ownership
flag mentioned above.
> I can post a patch for either of the above.  I welcome any other ideas from developers
to fix this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org


Mime
View raw message