hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1846) Don't fill preallocated portion of edits log with 0x00
Date Thu, 28 Apr 2011 05:35:03 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13026126#comment-13026126
] 

Todd Lipcon commented on HDFS-1846:
-----------------------------------

I ran tests on a non-SSD as well, and also added code to calculate the 99th percentile latency.
Here are my results:

{noformat}

Results for classic scheme:
Overall total ops: 10000
Overall total time of all ops: 103168.0
Overall average time of op: 10.3168
Overall fastest op: 5
Overall slowest op: 400
Preallocation total ops: 3
Preallocation total time of all ops: 96.0
Preallocation average time of op: 32.0
Preallocation fastest op: 31
Preallocation slowest op: 33
Total time of slowest 1% of ops: 8462.0
Average time of slowest 1% of ops: 84.62
99th percentile latency: 39


Results for new scheme:
Overall total ops: 10000
Overall total time of all ops: 91469.0
Overall average time of op: 9.1469
Overall fastest op: 2
Overall slowest op: 341
Preallocation total ops: 3
Preallocation total time of all ops: 251.0
Preallocation average time of op: 83.66666666666667
Preallocation fastest op: 49
Preallocation slowest op: 149
Total time of slowest 1% of ops: 6787.0
Average time of slowest 1% of ops: 67.87
99th percentile latency: 25
{noformat}

So, my results agree with Aaron's that this is an improvement for total throughput as well
as 99th percentile, even though the preallocations take longer. If we find at some point that
the preallocation is a performance issue, it wouldn't be *too* hard to push preallocation
to a separate thread. But let's not add that complexity now, since this patch seems to be
an overall improvement for durability as well as average performance.

> Don't fill preallocated portion of edits log with 0x00
> ------------------------------------------------------
>
>                 Key: HDFS-1846
>                 URL: https://issues.apache.org/jira/browse/HDFS-1846
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.23.0
>            Reporter: Aaron T. Myers
>            Assignee: Aaron T. Myers
>         Attachments: editsStored, hdfs-1846-perf-analysis.0.patch, hdfs-1846.0.txt, hdfs-1846.1.patch,
hdfs-1846.2.patch, hdfs-1846.3.patch, hdfs-1846.3.patch
>
>
> HADOOP-2330 added a feature to preallocate space in the local file system for the NN
transaction log. That change seeks past the current end of the file and writes out some data,
which on most systems results in the intervening data in the file being filled with zeros.
Most underlying file systems have special handling for sparse files, and don't actually allocate
blocks on disk for blocks of a file which consist completely of 0x00.
> I've seen cases in the wild where the volume an edits dir is on fills up, resulting in
a partial final transaction being written out to disk. If you examine the bytes of this (now
corrupt) edits file, you'll see the partial final transaction followed by a lot of zeros,
suggesting that the preallocation previously succeeded before the volume ran out of space.
If we fill the preallocated space with something other than zeros, we'd likely see the failure
at preallocation time, rather than transaction-writing time, and so cause the NN to crash
earlier, without a partial transaction being written out.
> I also hypothesize that filling the preallocated space in the edits log with something
other than 0x00 will result in a performance improvement in NN throughput. I haven't tested
this yet, but I intend to as part of this JIRA.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message