hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "heyongqiang" <heyongqi...@software.ict.ac.cn>
Subject Fw: Re: Should there be a way not maintaining the whole namespace structure in memory?
Date Tue, 01 Jul 2008 05:29:22 GMT
发件人: heyongqiang
发送时间: 2008-07-01 13:28:33
收件人: core-user
抄送: 
主题: Re: Should there be a way not maintaining the whole namespace structure in memory?

at now,i have not find a good way for maintainning the fsimage format. so, any suggestions?

thanks!




Best regards,
 
Yongqiang He
2008-07-01

Email: heyongqiang@software.ict.ac.cn
Tel:   86-10-62600966(O)
 
Research Center for Grid and Service Computing,
Institute of Computing Technology, 
Chinese Academy of Sciences
P.O.Box 2704, 100080, Beijing, China 



发件人: heyongqiang
发送时间: 2008-07-01 13:25:11
收件人: core-user
抄送: 
主题: Should there be a way not maintaining the whole namespace structure in memory?

In now's hdfs implementation,all INodeFile and INodeDirectory objects were loaded into memory,this
is done when setting up the  FSNameSpacs structure set up at namenode startup.
the namenode will analyze the fsimage file and edit log file. And if there are milllions of
files or directories how it can be handled?

I have done an exprements by making dirs,before i exprements:
[hdfs@vega2000 bin]$ ps -p 9122 -o rss,size,vsize,%mem
  RSS    SZ    VSZ %MEM
153648 1193868 1275340  3.7

after i creating 10000 directories, it turns:
[hdfs@vega2000 bin]$ ps -p 9122 -o rss,size,vsize,%mem
  RSS    SZ    VSZ %MEM
169084 1193868 1275340  4.0

I m trying to improve the fsimage file,so that namenode can locate and load the needed information
at need,and just like linux vfs,we can only obtain an inode cache.So this can avoid loading
the whole namespace structure at startup.




Best regards,
 
Yongqiang He
2008-07-01

Email: heyongqiang@software.ict.ac.cn
Tel:   86-10-62600966(O)
 
Research Center for Grid and Service Computing,
Institute of Computing Technology, 
Chinese Academy of Sciences
P.O.Box 2704, 100080, Beijing, China 
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message