hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lars Hofhansl (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-744) Support hsync in HDFS
Date Sat, 19 May 2012 07:54:06 GMT

     [ https://issues.apache.org/jira/browse/HDFS-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Lars Hofhansl updated HDFS-744:
-------------------------------

    Attachment: HDFS-744-trunk-v6.patch

New patch.

* only uses one flag
* SequenceFile.Writer implement Syncable.
* syncFS is deprecated.
* added metrics
* added test using metrics to verify behavior
* added replication test, to make sure the hsync is happening correctly on replicas.
* found a bug, where sometimes a sync is missed, happens when the currentPacket in hsync is
null. Fixed now.

Currently I count syncs of "out" and "checksumOut" (in BlockReceiver) as two sync events.
Would be hard to do otherwise, as both can be null resp, in which case no sync is happening.

In rare cases I found that the metrics on the DNs are not updated fast enough, and a test
fails. Looks like there are existing tests that also use metrics that have the same problem.
Not sure what to do about that.

                
> Support hsync in HDFS
> ---------------------
>
>                 Key: HDFS-744
>                 URL: https://issues.apache.org/jira/browse/HDFS-744
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>          Components: data-node, hdfs client
>            Reporter: Hairong Kuang
>            Assignee: Lars Hofhansl
>         Attachments: HDFS-744-trunk-v2.patch, HDFS-744-trunk-v3.patch, HDFS-744-trunk-v4.patch,
HDFS-744-trunk-v5.patch, HDFS-744-trunk-v6.patch, HDFS-744-trunk.patch, hdfs-744-v2.txt, hdfs-744-v3.txt,
hdfs-744.txt
>
>
> HDFS-731 implements hsync by default as hflush. As descriibed in HADOOP-6313, the real
expected semantics should be "flushes out to all replicas and all replicas have done posix
fsync equivalent - ie the OS has flushed it to the disk device (but the disk may have it in
its cache)." This jira aims to implement the expected behaviour.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message