hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2080) Speed up DFS read path
Date Mon, 20 Jun 2011 05:29:47 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Todd Lipcon updated HDFS-2080:
------------------------------

    Attachment: hdfs-2080.txt

Here's a combined patch to demonstrate the speed improvements.  It will need to be split up
into a few separate changes to be committed. Here's a summary of the changes in this somewhat
large patch:

bq. common/LICENSE.txt                                 |   20 +
- I borrowed some code from some BSD-licensed projects (hstore and "Slicing-by-8")

bq. common/bin/hadoop-config.sh                        |    8 +-
- Fixes a bug introduced with RPMs where it wouldn't find native code properly from within
the build dir.

bq.  common/build.xml                                   |    9 +
- adds javah for the new NativeCrc32 class

bq.  .../java/org/apache/hadoop/util/DataChecksum.java  |   64 +++-
- Adds new CHECKSUM_CRC32C type for the "CRC32C" polynomial which has hardware support.
- Adds a copyOf() to create a new DataChecksum given an existing instance.
- Generalizes some checks for CRC32 to now apply to all size-4 checksums.
- Adds new verifySums function, basically borrowed from FSInputChecker.java but operating
on ByteBuffer instead, and calling out to native code when available

bq. .../java/org/apache/hadoop/util/NativeCrc32.java   |   68 +++
- Small wrapper around the new native code

bq. .../org/apache/hadoop/util/PureJavaCrc32C.java     |  454 ++++++++++++++++
- copy of PureJavaCrc32 but for the new polynomial. Identical code but different tables.

bq. common/src/native/Makefile.am                      |    6 +-
- adds new C code for crc32

bq. .../src/org/apache/hadoop/util/NativeCrc32.c       |  149 ++++++
- implementation of verifySums using the native code

bq. .../src/native/src/org/apache/hadoop/util/crc32.h  |  133 +++++
- C implementations of "slicing-by-8" for CRC32 and CRC32C

bq. .../hadoop/util/crc32_zlib_polynomial_tables.h     |  552 ++++++++++++++++++++
bq. .../src/org/apache/hadoop/util/crc32c_tables.h     |  313 +++++++++++
- codegenned tables for the above algorithms

bq. .../native/src/org/apache/hadoop/util/x86_crc32c.h |  181 +++++++
- cpu-detection code to determine if SSE4.2 extensions are available
- implementations using the hardware crc32 operation for 32-bit and 64-bit

bq. .../org/apache/hadoop/util/TestPureJavaCrc32.java  |   14 +-
- improvements to generate Table for arbitrary polynomials

bq. hdfs/build.xml                                     |    1 +
- change which might be a bad idea, which lets hdfs pick up the native libraries from the
built common

bq. .../java/org/apache/hadoop/hdfs/BlockReader.java   |  330 +++++--------
- rewrites BlockReader to not inherit from FSInputChecker, thus making it much simpler
- now calls the "bulk verify" method in DataChecksum


bq. .../org/apache/hadoop/hdfs/DFSOutputStream.java    |    5 +-
- change default checksum to CRC32C

bq. .../org/apache/hadoop/hdfs/DFSInputStream.java     |    8 +-
bq. .../hadoop/hdfs/server/common/JspHelper.java       |    7 +-
- use IOUtils.readFully instead of the duplicate code from BlockReader

bq. .../hadoop/hdfs/server/datanode/BlockReceiver.java |    3 +-
- don't assume checksum is always CRC32 for partial chunk append

> Speed up DFS read path
> ----------------------
>
>                 Key: HDFS-2080
>                 URL: https://issues.apache.org/jira/browse/HDFS-2080
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs client
>    Affects Versions: 0.23.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.23.0
>
>         Attachments: hdfs-2080.txt
>
>
> I've developed a series of patches that speeds up the HDFS read path by a factor of about
2.5x (~300M/sec to ~800M/sec for localhost reading from buffer cache) and also will make it
easier to allow for advanced users (eg hbase) to skip a buffer copy. 

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message