hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-744) Support hsync in HDFS
Date Sat, 19 May 2012 22:19:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13279619#comment-13279619

Lars Hofhansl commented on HDFS-744:

Upon further inspection it turns out the issue with the metrics not being updated in time
is due to the fact the flushOrSync is not on the synchronous hsync or hflush path in BlockReceiver.receivePacket.
When hsync returns the DNs have all gotten the data and the request to flush and sync, but
they are not necessarily finished with it.

The same, BTW, is currently (without my patch) true to hflush. When hflush returns the DNs
have received the data in their buffers, but they may not be done flushing the data to the
OS buffers.

If we wanted to fix this (which would involve doing the flushOrSync in the PacketResponder
if the request is a client request, which inconveniently would also serialize the flush/syncs
across replicas), this should be a different jira, I think.

For now, to make the tests a bit more predictable I'll increment the sync counter before the
actual sync operation rather than afterwards.

> Support hsync in HDFS
> ---------------------
>                 Key: HDFS-744
>                 URL: https://issues.apache.org/jira/browse/HDFS-744
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Hairong Kuang
>            Assignee: Lars Hofhansl
>         Attachments: HDFS-744-trunk-v2.patch, HDFS-744-trunk-v3.patch, HDFS-744-trunk-v4.patch,
HDFS-744-trunk-v5.patch, HDFS-744-trunk-v6.patch, HDFS-744-trunk.patch, hdfs-744-v2.txt, hdfs-744-v3.txt,
> HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, the real
expected semantics should be "flushes out to all replicas and all replicas have done posix
fsync equivalent - ie the OS has flushed it to the disk device (but the disk may have it in
its cache)." This jira aims to implement the expected behaviour.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message