hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "guodongdong (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-6154) Improve the speed of saveNameSpace´╝îmaking HDFS restart and checkPoint faster
Date Thu, 27 Mar 2014 02:54:16 GMT

     [ https://issues.apache.org/jira/browse/HDFS-6154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

guodongdong updated HDFS-6154:
------------------------------

    Description: 
There are two stages when namenode saving namespace, serializes INode, calculates MD5 and
writes to disk.  Now, two stages are doing serially, For improvement,  one thread serializes
INode, and another thread  calculates MD5 and writes to disk. It doubles the speed of saving
namespace, Details are showed as below:

Test environment:
  only test namenode saving namespace, dfsadmin -saveNamespace
  machine: 144GB, Intel(R) Xeon(R) CPU  E5645  @ 2.40GHz, 12 cpu, Raid 5 SAS Disk,  jdk 1.7.0
 
||image size||before optimizing||after optimizing ||
|1.2GB|22sec|11sec|
|4.3GB|66sec|36sec|
|22GB|406sec|250sec|

  was:
There are two stage In namenode savenamespace,  serializing INode, calculate MD5 and write
to disk.  Now, two stage is doing serially, In this improvement, it is doing  parallel, one
thread do serializing INode, other thread do calculating MD5 and writing to disk, it double
speed of savenamespace, Detail is show in table:

Testing environment:
  only test namenode savenamespace, dfsadmin -saveNamespace
    machine: 144GB, Intel(R) Xeon(R) CPU  E5645  @ 2.40GHz, 12 cpu, Raid 5 SAS Disk,  jdk
1.7.0
 
||image size||before optimizing||after optimizing ||
|1.2GB|22sec|11sec|
|4.3GB|66sec|36sec|
|22GB|406sec|250sec|


> Improve the speed of saveNameSpace´╝îmaking HDFS restart and checkPoint faster
> ----------------------------------------------------------------------------
>
>                 Key: HDFS-6154
>                 URL: https://issues.apache.org/jira/browse/HDFS-6154
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>    Affects Versions: 2.3.0
>            Reporter: guodongdong
>         Attachments: HDFS-6154-new-patch
>
>
> There are two stages when namenode saving namespace, serializes INode, calculates MD5
and writes to disk.  Now, two stages are doing serially, For improvement,  one thread serializes
INode, and another thread  calculates MD5 and writes to disk. It doubles the speed of saving
namespace, Details are showed as below:
> Test environment:
>   only test namenode saving namespace, dfsadmin -saveNamespace
>   machine: 144GB, Intel(R) Xeon(R) CPU  E5645  @ 2.40GHz, 12 cpu, Raid 5 SAS Disk,  jdk
1.7.0
>  
> ||image size||before optimizing||after optimizing ||
> |1.2GB|22sec|11sec|
> |4.3GB|66sec|36sec|
> |22GB|406sec|250sec|



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message