hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stephen O'Donnell (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-14617) Improve fsimage load time by writing sub-sections to the fsimage index
Date Fri, 23 Aug 2019 15:30:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-14617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914350#comment-16914350
] 

Stephen O'Donnell commented on HDFS-14617:
------------------------------------------

In my testing on the 3.3 branch OIV etc do not have any issues with this change, but the ReverseXML
processor cannot create the parallel section headings, but that does not prevent the solution
working OK.

[~hexiaoqiao] You mentioned that OIV had issues with the 'parallel image' on 2.7 - did you
get a chance to see what error / trace it gives, so we can see how difficult it would be to
ensure OIV can read a parallel image OK?

> Improve fsimage load time by writing sub-sections to the fsimage index
> ----------------------------------------------------------------------
>
>                 Key: HDFS-14617
>                 URL: https://issues.apache.org/jira/browse/HDFS-14617
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Stephen O'Donnell
>            Assignee: Stephen O'Donnell
>            Priority: Major
>             Fix For: 3.3.0
>
>         Attachments: HDFS-14617.001.patch, ParallelLoading.svg, SerialLoading.svg, dirs-single.svg,
flamegraph.parallel.svg, flamegraph.serial.svg, inodes.svg
>
>
> Loading an fsimage is basically a single threaded process. The current fsimage is written
out in sections, eg iNode, iNode_Directory, Snapshots, Snapshot_Diff etc. Then at the end
of the file, an index is written that contains the offset and length of each section. The
image loader code uses this index to initialize an input stream to read and process each section.
It is important that one section is fully loaded before another is started, as the next section
depends on the results of the previous one.
> What I would like to propose is the following:
> 1. When writing the image, we can optionally output sub_sections to the index. That way,
a given section would effectively be split into several sections, eg:
> {code:java}
>    inode_section offset 10 length 1000
>      inode_sub_section offset 10 length 500
>      inode_sub_section offset 510 length 500
>      
>    inode_dir_section offset 1010 length 1000
>      inode_dir_sub_section offset 1010 length 500
>      inode_dir_sub_section offset 1010 length 500
> {code}
> Here you can see we still have the original section index, but then we also have sub-section
entries that cover the entire section. Then a processor can either read the full section in
serial, or read each sub-section in parallel.
> 2. In the Image Writer code, we should set a target number of sub-sections, and then
based on the total inodes in memory, it will create that many sub-sections per major image
section. I think the only sections worth doing this for are inode, inode_reference, inode_dir
and snapshot_diff. All others tend to be fairly small in practice.
> 3. If there are under some threshold of inodes (eg 10M) then don't bother with the sub-sections
as a serial load only takes a few seconds at that scale.
> 4. The image loading code can then have a switch to enable 'parallel loading' and a 'number
of threads' where it uses the sub-sections, or if not enabled falls back to the existing logic
to read the entire section in serial.
> Working with a large image of 316M inodes and 35GB on disk, I have a proof of concept
of this change working, allowing just inode and inode_dir to be loaded in parallel, but I
believe inode_reference and snapshot_diff can be make parallel with the same technique.
> Some benchmarks I have are as follows:
> {code:java}
> Threads   1     2     3     4 
> --------------------------------
> inodes    448   290   226   189 
> inode_dir 326   211   170   161 
> Total     927   651   535   488 (MD5 calculation about 100 seconds)
> {code}
> The above table shows the time in seconds to load the inode section and the inode_directory
section, and then the total load time of the image.
> With 4 threads using the above technique, we are able to better than half the load time
of the two sections. With the patch in HDFS-13694 it would take a further 100 seconds off
the run time, going from 927 seconds to 388, which is a significant improvement. Adding more
threads beyond 4 has diminishing returns as there are some synchronized points in the loading
code to protect the in memory structures.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message