hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2129) Simplify BlockReader to not inherit from FSInputChecker
Date Wed, 02 Nov 2011 21:09:32 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13142531#comment-13142531
] 

Todd Lipcon commented on HDFS-2129:
-----------------------------------

Here are some benchmarks from TestParallelRead, with number of iterations jacked up and 100%
random reads (instead of seq/random mix). I did the benchmarks on top of HDFS-2533 since otherwise
I was ending up blocked on getBlockFile everywhere. These benchmarks are also on top of HDFS-2130
(CRC32C as default)

The middle column is the new code without native libraries available. The third column is
with native code available, taking advantage of the SSE4.2 CRC32C implementation in trunk.

| Threads | Trunk  | HDFS-2533              | HDFS-2533 + HDFS-2129 (nonative)  | HDFS-2553
+ HDFS-2129 + native |
| 4 | 226556 KB/s  | 236065 KB/sec (1.04x) |  231979 KB/sec (1.02x)            | 285824 KB/sec
(1.26x) |
| 16 | 377474 KB/s | 454362 KB/sec (1.20x) |  457497 KB/sec (1.20x)            | 526224 KB/sec
(1.39x) |
| 8 | 410114 KB/s  | 453107 KB/sec (1.10x) |  447927 KB/sec (1.02x)            | 549027 KB/sec
(1.33x) |

Further gains will come when HDFS-1148 is finished -- the 16-thread test in particular ends
up with a lot of contention on the FSDataset lock.
                
> Simplify BlockReader to not inherit from FSInputChecker
> -------------------------------------------------------
>
>                 Key: HDFS-2129
>                 URL: https://issues.apache.org/jira/browse/HDFS-2129
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs client, performance
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>             Fix For: 0.24.0
>
>         Attachments: hdfs-2129-benchmark.png, hdfs-2129.txt, hdfs-2129.txt, hdfs-2129.txt,
hdfs-2129.txt, hdfs-2129.txt, hdfs-2129.txt, hdfs-2129.txt, seq-read-1gb-bench.png
>
>
> BlockReader is currently quite complicated since it has to conform to the FSInputChecker
inheritance structure. It would be much simpler to implement it standalone. Benchmarking indicates
it's slightly faster, as well.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message