hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-12326) Implement ChecksumFileSystem#getFileChecksum equivalent to HDFS for easy check
Date Wed, 19 Aug 2015 01:41:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-12326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14702287#comment-14702287
] 

Hadoop QA commented on HADOOP-12326:
------------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch |  19m 55s | Pre-patch trunk compilation is healthy. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags.
|
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to include 3 new
or modified test files. |
| {color:green}+1{color} | javac |   7m 57s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |   9m 48s | There were no new javadoc warning messages.
|
| {color:green}+1{color} | release audit |   0m 23s | The applied patch does not increase
the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle |   2m  2s | The applied patch generated  3 new checkstyle
issues (total was 372, now 369). |
| {color:red}-1{color} | whitespace |   0m  1s | The patch has 1  line(s) that end in whitespace.
Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 23s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 33s | The patch built with eclipse:eclipse.
|
| {color:green}+1{color} | findbugs |   5m 13s | The patch does not introduce any new Findbugs
(version 3.0.0) warnings. |
| {color:red}-1{color} | common tests |  22m 58s | Tests failed in hadoop-common. |
| {color:green}+1{color} | tools/hadoop tests |   6m 15s | Tests passed in hadoop-streaming.
|
| {color:red}-1{color} | hdfs tests | 180m 56s | Tests failed in hadoop-hdfs. |
| | | 257m 35s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.ha.TestZKFailoverController |
|   | hadoop.net.TestNetUtils |
|   | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
|   | hadoop.hdfs.server.namenode.TestFsckWithMultipleNameNodes |
|   | hadoop.hdfs.server.namenode.TestFSDirectory |
|   | hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication |
|   | hadoop.hdfs.server.namenode.TestFileTruncate |
|   | hadoop.hdfs.server.namenode.TestFSNamesystem |
|   | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| Timed out tests | org.apache.hadoop.cli.TestHDFSCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12751087/HADOOP-12326.003.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 71aedfa |
| checkstyle |  https://builds.apache.org/job/PreCommit-HADOOP-Build/7501/artifact/patchprocess/diffcheckstylehadoop-common.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HADOOP-Build/7501/artifact/patchprocess/whitespace.txt
|
| hadoop-common test log | https://builds.apache.org/job/PreCommit-HADOOP-Build/7501/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-streaming test log | https://builds.apache.org/job/PreCommit-HADOOP-Build/7501/artifact/patchprocess/testrun_hadoop-streaming.txt
|
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HADOOP-Build/7501/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results | https://builds.apache.org/job/PreCommit-HADOOP-Build/7501/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HADOOP-Build/7501/console |


This message was automatically generated.

> Implement ChecksumFileSystem#getFileChecksum equivalent to HDFS for easy check
> ------------------------------------------------------------------------------
>
>                 Key: HADOOP-12326
>                 URL: https://issues.apache.org/jira/browse/HADOOP-12326
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: fs
>    Affects Versions: 2.7.1
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: HADOOP-12326.001.patch, HADOOP-12326.002.patch, HADOOP-12326.003.patch
>
>
> If we have same-content files, one local and one remotely on HDFS (after downloading
or uploading), getFileChecksum can provide a quick check whether they are consistent.  To
this end, we can switch to CRC32C on local filesystem. The difference in block sizes does
not matter, because for the local filesystem it's just a logical parameter.
> {code}
> $ hadoop fs -Dfs.local.block.size=134217728 -checksum file:${PWD}/part-m-00000 part-m-00000
> 15/08/15 13:30:02 WARN util.NativeCodeLoader: Unable to load native-hadoop library for
your platform... using builtin-java classes where applicable
> file:///Users/gshegalov/workspace/hadoop-common/part-m-00000	MD5-of-262144MD5-of-512CRC32C
000002000000000000040000e84fb07f8c9d4ef3acb5d1983a7e2a68
> part-m-00000	MD5-of-262144MD5-of-512CRC32C	000002000000000000040000e84fb07f8c9d4ef3acb5d1983a7e2a68
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message