hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-10681) Remove synchronized blocks from SnappyCodec and ZlibCodec buffering inner loop
Date Mon, 06 Oct 2014 15:11:35 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-10681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160380#comment-14160380
] 

Hudson commented on HADOOP-10681:
---------------------------------

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1918 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1918/])
HADOOP-10681. Remove unnecessary synchronization from Snappy & Zlib codecs. Contributed
by Gopal Vijayaraghavan. (acmurthy: rev 8f9ab998e273259c1e7a3ed53ba37d767e02b6bb)
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibDecompressor.java
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/zlib/ZlibCompressor.java
* hadoop-common-project/hadoop-common/CHANGES.txt
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyDecompressor.java
* hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/compress/snappy/SnappyCompressor.java
* hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/io/compress/TestCodecPool.java


> Remove synchronized blocks from SnappyCodec and ZlibCodec buffering inner loop
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-10681
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10681
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: performance
>    Affects Versions: 2.2.0, 2.4.0, 2.5.0
>            Reporter: Gopal V
>            Assignee: Gopal V
>              Labels: perfomance
>             Fix For: 2.6.0
>
>         Attachments: HADOOP-10681.1.patch, HADOOP-10681.2.patch, HADOOP-10681.3.patch,
HADOOP-10681.4.patch, compress-cmpxchg-small.png, perf-top-spill-merge.png, snappy-perf-unsync.png
>
>
> The current implementation of SnappyCompressor spends more time within the java loop
of copying from the user buffer into the direct buffer allocated to the compressor impl, than
the time it takes to compress the buffers.
> !perf-top-spill-merge.png!
> The bottleneck was found to be java monitor code inside SnappyCompressor.
> The methods are neatly inlined by the JIT into the parent caller (BlockCompressorStream::write),
which unfortunately does not flatten out the synchronized blocks.
> !compress-cmpxchg-small.png!
> The loop does a write of small byte[] buffers (each IFile key+value). 
> I counted approximately 6 monitor enter/exit blocks per k-v pair written.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message