hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Wei-Chiu Chuang <weic...@cloudera.com>
Subject [Erasure Coding] Java-based codec does not seem to reconstruct strips correctly
Date Fri, 21 Oct 2016 11:43:16 GMT
Fellow HDFS developers,
Want to bring this issue up to your attention: HDFS-10935 <https://issues.apache.org/jira/browse/HDFS-10935>
(Java-based EC codec does not reconstruct blocks correctly)

Initially I found 9 out of 30 TestFileChecksum tests failed consistently on my Mac IntelliJ.
More specifically, they failed with error “Checksum mismatch”.
Because jenkins job never picked up this failure, I thought it must have been some kind of
environment issue.
However, after consulting a number of fellow HDFS developers (Kai, Sammi, Eddy, Sean, Andrew
and John), it turns out that this set of tests fail without native ISA-L library.

I was able to set up two local Hadoop repo on a Linux box, one is built with native ISA-L
lib and the other isn’t. And the one with native lib always succeeds and the other always

If I understand them correctly, these tests verifies that blocks are reconstructed correctly
after one datanode is shutdown, and therefore, the checksum mismatch w/o native ISA-L lib
seems to suggest that Java-based EC codec does not reconstruct strips correctly. I do not
have sufficient insight into EC reconstruction so I maybe wrong to interpret the failures
this way. But I think this is severe enough to become a blocker for the next Hadoop 3 alpha

Wei-Chiu Chuang
A very happy Clouderan

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message