hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-895) Allow hflush/sync to occur in parallel with new writes to the file
Date Mon, 15 Nov 2010 16:01:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932093#action_12932093
] 

Todd Lipcon commented on HDFS-895:
----------------------------------

With this latest patch, the following tests fali:
    [junit] Test org.apache.hadoop.hdfs.TestFileStatus FAILED [ HDFS-1470 ]
    [junit] Test org.apache.hadoop.hdfs.TestHDFSTrash FAILED (timeout) [ HDFS-1471 ]
    [junit] Test org.apache.hadoop.hdfs.server.namenode.TestStorageRestore FAILED [ HDFS-1496
]
    [junit] Test org.apache.hadoop.fs.TestHDFSFileContextMainOperations FAILED [ HDFS-874
]
    [junit] Test org.apache.hadoop.hdfs.TestPipelines FAILED [ HDFS-1467 ]
    [junit] Test org.apache.hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewer FAILED
 [ HDFS-1500 ]
All of these also fail in trunk - I put the relevant JIRAs next to them above. So, no new
failures caused by this patch.

Test-patch results:

     [exec] -1 overall.  
     [exec] 
     [exec]     +1 @author.  The patch does not contain any @author tags.
     [exec] 
     [exec]     +1 tests included.  The patch appears to include 5 new or modified tests.
     [exec] 
     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.
     [exec] 
     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler
warnings.
     [exec] 
     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.
     [exec] 
     [exec]     -1 release audit.  The applied patch generated 97 release audit warnings (more
than the trunk's current 1 warnings).
     [exec] 
     [exec]     +1 system test framework.  The patch passed system test framework compile.

The release audit is incorrect - same is true of an empty patch on HDFS trunk (known issue)

> Allow hflush/sync to occur in parallel with new writes to the file
> ------------------------------------------------------------------
>
>                 Key: HDFS-895
>                 URL: https://issues.apache.org/jira/browse/HDFS-895
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: hdfs client
>    Affects Versions: 0.22.0
>            Reporter: dhruba borthakur
>            Assignee: Todd Lipcon
>             Fix For: 0.22.0
>
>         Attachments: 895-delta-for-review.txt, hdfs-895-0.20-append.txt, hdfs-895-20.txt,
hdfs-895-branch-20-append.txt, hdfs-895-ontopof-1497.txt, hdfs-895-review.txt, hdfs-895-trunk.txt,
hdfs-895.txt, hdfs-895.txt, hdfs-895.txt, hdfs-895.txt, hdfs-895.txt, hdfs-895.txt, hdfs-895.txt
>
>
> In the current trunk, the HDFS client methods writeChunk() and hflush./sync are syncronized.
This means that if a hflush/sync is in progress, an applicationn cannot write data to the
HDFS client buffer. This reduces the write throughput of the transaction log in HBase. 
> The hflush/sync should allow new writes to happen to the HDFS client even when a hflush/sync
is in progress. It can record the seqno of the message for which it should receice the ack,
indicate to the DataStream thread to star flushing those messages, exit the synchronized section
 and just wai for that ack to arrive.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message