hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6109) let sync_file_range() system call run in background
Date Wed, 28 May 2014 21:55:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14011663#comment-14011663
] 

Andrew Wang commented on HDFS-6109:
-----------------------------------

Maybe we can reuse the FsDatasetAsyncDiskService available in FsDatasetImpl? Right now it's
used to do async deletes, but it seems apt to do this sort of thing there too. It'd be good
so we don't overload a single volume with different types of I/O work.

> let sync_file_range() system call run in background
> ---------------------------------------------------
>
>                 Key: HDFS-6109
>                 URL: https://issues.apache.org/jira/browse/HDFS-6109
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 3.0.0, 2.3.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>         Attachments: HDFS-6109-v2.txt, HDFS-6109.txt
>
>
> Through we passed SYNC_FILE_RANGE_WRITE to sync_file_range, to make it as asynchronous
as possible, it still could be blocked, e.g. the os io request queue is full.
> Since we use sync_file_range just as a page cache advisor role:) it doesn't decide or
guarantee the real durability, it would be nice if we could run it in  backgroud. At least
my test log showed, a few sync_file_range calls still cost tens of ms or more, due to the
happened location is in the critical write path(BlockReceiver class), from a upper view, like
HBase application, will "hung" tens of ms as well during Hlog syncing.
> Generally speaking, the patch could not improve too much, but, better than before, right
? :)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message