hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Liang Xie (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-6109) let sync_file_range() system call run in backgroud
Date Mon, 26 May 2014 09:32:02 GMT

     [ https://issues.apache.org/jira/browse/HDFS-6109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Liang Xie updated HDFS-6109:
----------------------------

    Attachment: HDFS-6109.txt

Attached is a straight-forward trunk patch, please help to review, thanks.
I introduced a config per comment and the default behivor keeps the same as before.
This change is only meaningful for latency sensitive application like HBase. Every write stall
from datanode layer will let HBase WAL sync operation hung the similar time range at least.
I found this post from one fb mysql developer is really nice:
http://yoshinorimatsunobu.blogspot.hk/2014/03/how-syncfilerange-really-works.html
Especially the final "Solution for the stalls" section:
{code}
calling sync_file_range() from background threads (not from user-facing thread) are important
{code}

> let sync_file_range() system call run in backgroud
> --------------------------------------------------
>
>                 Key: HDFS-6109
>                 URL: https://issues.apache.org/jira/browse/HDFS-6109
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 3.0.0, 2.3.0
>            Reporter: Liang Xie
>            Assignee: Liang Xie
>         Attachments: HDFS-6109.txt
>
>
> Through we passed SYNC_FILE_RANGE_WRITE to sync_file_range, to make it as asynchronous
as possible, it still could be blocked, e.g. the os io request queue is full.
> Since we use sync_file_range just as a page cache advisor role:) it doesn't decide or
guarantee the real durability, it would be nice if we could run it in  backgroud. At least
my test log showed, a few sync_file_range calls still cost tens of ms or more, due to the
happened location is in the critical write path(BlockReceiver class), from a upper view, like
HBase application, will "hung" tens of ms as well during Hlog syncing.
> Generally speaking, the patch could not improve too much, but, better than before, right
? :)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message