hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "genericqa (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-6852) apparent bug in concatenated-bzip2 support (decoding)
Date Tue, 06 Feb 2018 13:35:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16353881#comment-16353881
] 

genericqa commented on HADOOP-6852:
-----------------------------------

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue}  0m 24s{color} | {color:blue}
Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green}  0m  0s{color} |
{color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m  0s{color}
| {color:green} The patch appears to include 7 new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  1m 27s{color} | {color:blue}
Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 31s{color}
| {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 18m 23s{color} | {color:red}
root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green}  2m 16s{color}
| {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 13s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 26s{color}
| {color:green} branch has no errors when building and testing our client artifacts. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  0s{color} | {color:blue}
Skipped patched modules with no Java source: hadoop-client-modules/hadoop-client-minicluster
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  1m 52s{color} |
{color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 22s{color} |
{color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue}  0m 16s{color} | {color:blue}
Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green}  4m 29s{color}
| {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m 57s{color} |
{color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 17m 57s{color} | {color:red}
root generated 183 new + 1054 unchanged - 0 fixed = 1237 total (was 1054) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}  2m 24s{color}
| {color:orange} root: The patch generated 16 new + 71 unchanged - 9 fixed = 87 total (was
80) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green}  2m 34s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green}  0m  0s{color}
| {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green}  0m  1s{color} | {color:green}
The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 25s{color}
| {color:green} patch has no errors when building and testing our client artifacts. {color}
|
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue}  0m  0s{color} | {color:blue}
Skipped patched modules with no Java source: hadoop-client-modules/hadoop-client-minicluster
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green}  2m 34s{color} |
{color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green}  1m 50s{color} |
{color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  9m 18s{color} | {color:green}
hadoop-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}130m 15s{color} | {color:red}
hadoop-mapreduce-client-jobclient in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m  3s{color} | {color:green}
hadoop-client-minicluster in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green}  0m 56s{color}
| {color:green} The patch does not generate ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}243m 49s{color} | {color:black}
{color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.v2.TestUberAM |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | HADOOP-6852 |
| JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12909398/HADOOP-6852.02.patch
|
| Optional Tests |  asflicense  compile  javac  javadoc  mvninstall  mvnsite  unit  shadedclient
 xml  findbugs  checkstyle  |
| uname | Linux fb14dce7c82a 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18 11:55:51 UTC 2017
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 87429af |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| compile | https://builds.apache.org/job/PreCommit-HADOOP-Build/14076/artifact/out/branch-compile-root.txt
|
| findbugs | v3.1.0-RC1 |
| javac | https://builds.apache.org/job/PreCommit-HADOOP-Build/14076/artifact/out/diff-compile-javac-root.txt
|
| checkstyle | https://builds.apache.org/job/PreCommit-HADOOP-Build/14076/artifact/out/diff-checkstyle-root.txt
|
| unit | https://builds.apache.org/job/PreCommit-HADOOP-Build/14076/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
|  Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/14076/testReport/ |
| Max. process+thread count | 1363 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient
hadoop-client-modules/hadoop-client-minicluster U: . |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/14076/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT   http://yetus.apache.org |


This message was automatically generated.



> apparent bug in concatenated-bzip2 support (decoding)
> -----------------------------------------------------
>
>                 Key: HADOOP-6852
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6852
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: io
>    Affects Versions: 0.22.0
>         Environment: Linux x86_64 running 32-bit Hadoop, JDK 1.6.0_15
>            Reporter: Greg Roelofs
>            Assignee: Zsolt Venczel
>            Priority: Major
>         Attachments: HADOOP-6852.01.patch, HADOOP-6852.02.patch
>
>
> The following simplified code (manually picked out of testMoreBzip2() in https://issues.apache.org/jira/secure/attachment/12448272/HADOOP-6835.v4.trunk-hadoop-mapreduce.patch)
triggers a "java.io.IOException: bad block header" in org.apache.hadoop.io.compress.bzip2.CBZip2InputStream.initBlock(
CBZip2InputStream.java:527):
> {noformat}
>     JobConf jobConf = new JobConf(defaultConf);
>     CompressionCodec bzip2 = new BZip2Codec();
>     ReflectionUtils.setConf(bzip2, jobConf);
>     localFs.delete(workDir, true);
>     // copy multiple-member test file to HDFS
>     String fn2 = "testCompressThenConcat.txt" + bzip2.getDefaultExtension();
>     Path fnLocal2 = new Path(System.getProperty("test.concat.data","/tmp"),fn2);
>     Path fnHDFS2  = new Path(workDir, fn2);
>     localFs.copyFromLocalFile(fnLocal2, fnHDFS2);
>     FileInputFormat.setInputPaths(jobConf, workDir);
>     final FileInputStream in2 = new FileInputStream(fnLocal2.toString());
>     CompressionInputStream cin2 = bzip2.createInputStream(in2);
>     LineReader in = new LineReader(cin2);
>     Text out = new Text();
>     int numBytes, totalBytes=0, lineNum=0;
>     while ((numBytes = in.readLine(out)) > 0) {
>       ++lineNum;
>       totalBytes += numBytes;
>     }
>     in.close();
> {noformat}
> The specified file is also included in the H-6835 patch linked above, and some additional
debug output is included in the commented-out test loop above.  (Only in the linked, "v4"
version of the patch, however--I'm about to remove the debug stuff for checkin.)
> It's possible I've done something completely boneheaded here, but the file, at least,
checks out in a subsequent set of subtests and with stock bzip2 itself.  Only the code above
is problematic; it reads through the first concatenated chunk (17 lines of text) just fine
but chokes on the header of the second one.  Altogether, the test file contains 84 lines of
text and 4 concatenated bzip2 files.
> (It's possible this is a mapreduce issue rather than common, but note that the identical
gzip test works fine.  Possibly it's related to the stream-vs-decompressor dichotomy, though;
intentionally not supported?)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message