hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "heyongqiang" <heyongqi...@software.ict.ac.cn>
Subject Should there be a way not maintaining the whole namespace structure in memory?
Date Tue, 01 Jul 2008 05:25:11 GMT
In now's hdfs implementation,all INodeFile and INodeDirectory objects were loaded into memory,this
is done when setting up the  FSNameSpacs structure set up at namenode startup.
the namenode will analyze the fsimage file and edit log file. And if there are milllions of
files or directories how it can be handled?

I have done an exprements by making dirs,before i exprements:
[hdfs@vega2000 bin]$ ps -p 9122 -o rss,size,vsize,%mem
  RSS    SZ    VSZ %MEM
153648 1193868 1275340  3.7

after i creating 10000 directories, it turns:
[hdfs@vega2000 bin]$ ps -p 9122 -o rss,size,vsize,%mem
  RSS    SZ    VSZ %MEM
169084 1193868 1275340  4.0

I m trying to improve the fsimage file,so that namenode can locate and load the needed information
at need,and just like linux vfs,we can only obtain an inode cache.So this can avoid loading
the whole namespace structure at startup.

Best regards,
Yongqiang He

Email: heyongqiang@software.ict.ac.cn
Tel:   86-10-62600966(O)
Research Center for Grid and Service Computing,
Institute of Computing Technology, 
Chinese Academy of Sciences
P.O.Box 2704, 100080, Beijing, China 

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message