Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Sun, 14 Apr 2013 22:54:15 +0000 (UTC)
From: "Ivan Mitic (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12641530.1365490628792.165363.1365980055679@arcas>
In-Reply-To: <JIRA.12641530.1365490628792@arcas>
References: <JIRA.12641530.1365490628792@arcas>
Subject: [jira] [Updated] (HDFS-4677) Editlog should support synchronous
 writes
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


     [ https://issues.apache.org/jira/browse/HDFS-4677?page=3Dcom.atlassian=
.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ivan Mitic updated HDFS-4677:
-----------------------------

    Attachment: HDFS-4677.patch

Attaching the patch. Will also attach the branch-1 compatible patch once th=
is one is reviewed.

I see that {{EditLogFileOutputStream}} already has a {{shouldSkipFsyncForTe=
sts}} flag which is used to speed up test execution. I did a quick experime=
nt on my Ubuntu box to see if we can remove this test hook in favor of the =
newly introduced config and still keep the test runtime reasonably low. Tur=
ned out that we cannot. TestEditLog which run ~3 minutes took more than 15 =
minutes with the new config (it timed out actually), so keeping the existin=
g logic as-is.
               =20
> Editlog should support synchronous writes
> -----------------------------------------
>
>                 Key: HDFS-4677
>                 URL: https://issues.apache.org/jira/browse/HDFS-4677
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 3.0.0, 1-win
>            Reporter: Ivan Mitic
>            Assignee: Ivan Mitic
>         Attachments: HDFS-4677.patch
>
>
> In the current implementation, NameNode editlog performs syncs to the per=
sistent storage using the {{FileChannel#force}} Java APIs. This API is docu=
mented to be slower compared to an alternative where {{RandomAccessFile}} i=
s opened with "rws" flags (synchronous writes).=20
> We instrumented {{FileChannel#force}} on Windows and it some software/har=
dware configurations it can perform significantly slower than the =E2=80=9C=
rws=E2=80=9D alternative.
> In terms of the Windows APIs, FileChannel#force internally calls [FlushFi=
leBuffers|http://msdn.microsoft.com/en-us/library/windows/desktop/aa364439(=
v=3Dvs.85).aspx] while RandomAccessFile (=E2=80=9Crws=E2=80=9D) opens the f=
ile with the [FILE_FLAG_WRITE_THROUGH flag|http://support.microsoft.com/kb/=
99794].=20
> With this Jira I'd like to introduce a flag that provide means to configu=
re NameNode to use synchronous writes. There is a catch though, the behavio=
r of the "rws" flags is platform and hardware specific and might not provid=
e the same level of guarantees as {{FileChannel#force}} w.r.t. flushing the=
 on-disk cache. This is an expert level setting, and it should be documente=
d as such.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrato=
rs
For more information on JIRA, see: http://www.atlassian.com/software/jira