hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9092) Nfs silently drops overlapping write requests and causes data copying to fail
Date Tue, 29 Sep 2015 01:01:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14934434#comment-14934434
] 

Hadoop QA commented on HDFS-9092:
---------------------------------

\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch |  18m 38s | Pre-patch trunk has 1 extant Findbugs (version
3.0.0) warnings. |
| {color:green}+1{color} | @author |   0m  0s | The patch does not contain any @author tags.
|
| {color:green}+1{color} | tests included |   0m  0s | The patch appears to include 1 new
or modified test files. |
| {color:green}+1{color} | javac |   7m 57s | There were no new javac warning messages. |
| {color:green}+1{color} | javadoc |  10m 23s | There were no new javadoc warning messages.
|
| {color:red}-1{color} | release audit |   0m 20s | The applied patch generated 1 release
audit warnings. |
| {color:green}+1{color} | checkstyle |   1m 47s | There were no new checkstyle issues. |
| {color:red}-1{color} | whitespace |   0m  1s | The patch has 3  line(s) that end in whitespace.
Use git apply --whitespace=fix. |
| {color:green}+1{color} | install |   1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse |   0m 34s | The patch built with eclipse:eclipse.
|
| {color:red}-1{color} | findbugs |   3m 23s | The patch appears to introduce 2 new Findbugs
(version 3.0.0) warnings. |
| {color:green}+1{color} | native |   3m 13s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 162m 40s | Tests failed in hadoop-hdfs. |
| {color:green}+1{color} | hdfs tests |   1m 47s | Tests passed in hadoop-hdfs-nfs. |
| | | 212m 19s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs-nfs |
| Failed unit tests | hadoop.hdfs.server.namenode.TestStartupProgressServlet |
|   | hadoop.hdfs.server.namenode.TestFileTruncate |
|   | hadoop.hdfs.tools.TestDFSAdminWithHA |
|   | hadoop.hdfs.web.TestWebHDFSOAuth2 |
|   | hadoop.hdfs.server.namenode.TestCacheDirectives |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL | http://issues.apache.org/jira/secure/attachment/12764084/HDFS-9092.002.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / e5992ef |
| Pre-patch Findbugs warnings | https://builds.apache.org/job/PreCommit-HDFS-Build/12719/artifact/patchprocess/trunkFindbugsWarningshadoop-hdfs.html
|
| Release Audit | https://builds.apache.org/job/PreCommit-HDFS-Build/12719/artifact/patchprocess/patchReleaseAuditProblems.txt
|
| whitespace | https://builds.apache.org/job/PreCommit-HDFS-Build/12719/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings | https://builds.apache.org/job/PreCommit-HDFS-Build/12719/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs-nfs.html
|
| hadoop-hdfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/12719/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| hadoop-hdfs-nfs test log | https://builds.apache.org/job/PreCommit-HDFS-Build/12719/artifact/patchprocess/testrun_hadoop-hdfs-nfs.txt
|
| Test Results | https://builds.apache.org/job/PreCommit-HDFS-Build/12719/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf900.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep
3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-HDFS-Build/12719/console |


This message was automatically generated.

> Nfs silently drops overlapping write requests and causes data copying to fail
> -----------------------------------------------------------------------------
>
>                 Key: HDFS-9092
>                 URL: https://issues.apache.org/jira/browse/HDFS-9092
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: nfs
>    Affects Versions: 2.7.1
>            Reporter: Yongjun Zhang
>            Assignee: Yongjun Zhang
>         Attachments: HDFS-9092.001.patch, HDFS-9092.002.patch
>
>
> When NOT using 'sync' option, the NFS writes may issue the following warning:
> org.apache.hadoop.hdfs.nfs.nfs3.OpenFileCtx: Got an overlapping write (1248751616, 1249677312),
nextOffset=1248752400. Silently drop it now
> and the size of data copied via NFS will stay at 1248752400.
> Found what happened is:
> 1. The write requests from client are sent asynchronously. 
> 2. The NFS gateway has handler to handle the incoming requests by creating an internal
write request structuire and put it into cache;
> 3. In parallel, a separate thread in NFS gateway takes requests out from the cache and
writes the data to HDFS.
> The current offset is how much data has been written by the write thread in 3. The detection
of overlapping write request happens in 2, but it only checks the write request against the
curent offset, and trim the request if necessary. Because the write requests are sent asynchronously,
if two requests are beyond the current offset, and they overlap, it's not detected and both
are put into the cache. This cause the symptom reported in this case at step 3.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message