hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HADOOP-11996) Native erasure coder facilities based on ISA-L
Date Tue, 02 Feb 2016 08:44:39 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-11996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kai Zheng updated HADOOP-11996:
-------------------------------
    Attachment: HADOOP-11996-v7.patch

Rebased the patch:
1. Building with {{mvn package -Pdist -Pnative -Drequire.isal -Dbundle.isal -Disal.lib=/usr/lib
-DskipTests}}
{noformat}
...
[INFO] Apache Hadoop Mini-Cluster ........................ SUCCESS [  0.050 s]
[INFO] Apache Hadoop Scheduler Load Simulator ............ SUCCESS [  3.277 s]
[INFO] Apache Hadoop Tools Dist .......................... SUCCESS [  2.738 s]
[INFO] Apache Hadoop Kafka Library support ............... SUCCESS [  1.460 s]
[INFO] Apache Hadoop Tools ............................... SUCCESS [  0.020 s]
[INFO] Apache Hadoop Distribution ........................ SUCCESS [ 16.572 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 11:38 min
[INFO] Finished at: 2016-02-03T16:25:44+08:00
[INFO] Final Memory: 164M/896M
[INFO] ------------------------------------------------------------------------
{noformat}
2. Checknative
{noformat}
[root@zkdesk hadoop]# cd hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/
[root@zkdesk hadoop-3.0.0-SNAPSHOT]# bin/hadoop checknative
16/02/03 16:26:23 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2
library system-native
16/02/03 16:26:23 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib
library
Native library checking:
hadoop:  true /home/workspace/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
snappy:  true /lib64/libsnappy.so.1
lz4:     true revision:99
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   true /home/workspace/hadoop/hadoop-dist/target/hadoop-3.0.0-SNAPSHOT/lib/native/libisal.so.2
{noformat}
3. Run the test
{noformat}
[root@zkdesk hadoop]# ./hadoop-common-project/hadoop-common/target/native/erasure_code_test
Performing erasure code test
Recovering (numAllUnits = 9, numDataUnits = 6, numErased = 2)
 - ErasedIndexes =  1 7       - DecodeIndex =  2 3 4 5 6 8

EncodeMatrix:
 1 0 0 0 0 0 0 1 0
 0 0 0 0 0 1 0 0 0
 0 0 0 1 0 0 0 0 0
 0 1 0 0 0 0 0 0 1
 122 186 71 167 142 244 186 122 167
 71 244 142 173 157 221 152 61 170

InvertMatrix:
 2 173 10 213 24 40
 62 183 132 127 27 36
 1 0 0 0 0 0
 0 1 0 0 0 0
 0 0 1 0 0 0
 0 0 0 1 0 0

DecodeMatrix:
 62 183 132 127 27 36 48 175 26
 24 92 105 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0
 0 0 0 0 0 0 0 0 0

Successfully done, passed!
{noformat}
4. Building on Windows passed.


> Native erasure coder facilities based on ISA-L
> ----------------------------------------------
>
>                 Key: HADOOP-11996
>                 URL: https://issues.apache.org/jira/browse/HADOOP-11996
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: io
>            Reporter: Kai Zheng
>            Assignee: Kai Zheng
>         Attachments: HADOOP-11996-initial.patch, HADOOP-11996-v2.patch, HADOOP-11996-v3.patch,
HADOOP-11996-v4.patch, HADOOP-11996-v5.patch, HADOOP-11996-v6.patch, HADOOP-11996-v7.patch
>
>
> While working on HADOOP-11540 and etc., it was found useful to write the basic facilities
based on Intel ISA-L library separately from JNI stuff. It's also easy to debug and troubleshooting,
as no JNI or Java stuffs are involved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message