hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Pinski (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-11660) Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
Date Sat, 13 Jun 2015 12:01:01 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14584577#comment-14584577
] 

Andrew Pinski commented on HADOOP-11660:
----------------------------------------

> On aarch64 a crc32 takes 3 exec cycles, but it only has one crc unit, so if
> we did the same on aarch64 and had say
> crc32 w0, w0, x3
> crc32 w1, w1, x4
> crc32 w2, w2, x5
> The second crc32 w1, w1, x4 would have to wait for the 1st crc to complete
> and the 3rd would have to wait for the 2nd, taking 9 cycles in any case, so
> there is no benefit to pipelining.

This is not true on some AARCH64 processors.  For ThunderX, this is definitely not true. 
crc32 (32bits) is fully pipelined and the next one will issue right away.
Though the latency of those instructions are 4 cycles long.  So this will only take 6 cycles
on ThunderX.

Thanks,
Andrew Pinski

> Add support for hardware crc of HDFS checksums on ARM aarch64 architecture
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-11660
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11660
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: native
>    Affects Versions: 2.8.0
>         Environment: ARM aarch64 development platform
>            Reporter: Edward Nevill
>            Assignee: Edward Nevill
>            Priority: Minor
>              Labels: performance
>             Fix For: 2.8.0
>
>         Attachments: jira-11660.patch
>
>   Original Estimate: 48h
>  Remaining Estimate: 48h
>
> This patch adds support for hardware crc for ARM's new 64 bit architecture
> The patch is completely conditionalized on __aarch64__
> I have only added support for the non pipelined version as I benchmarked the pipelined
version on aarch64 and it showed no performance improvement.
> The aarch64 version supports both Castagnoli and Zlib CRCs as both of these are supported
on ARM aarch64 hardwre.
> To benchmark this I modified the test_bulk_crc32 test to print out the time taken to
CRC a 1MB dataset 1000 times.
> Before:
> CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
> CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 2.55
> After:
> CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
> CRC 1048576 bytes @ 512 bytes per checksum X 1000 iterations = 0.57
> So this represents a 5X performance improvement on raw CRC calculation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message