hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Manish Shah <man...@rapleaf.com>
Subject NameNode hardware specs
Date Tue, 12 Aug 2008 18:24:45 GMT
Can someone help explain in a little more detail some of the reasons  
for the hardware specs that were recently added to the wiki for the  
NameNode.  I guess i'm interested in learning how others have settled  
on these specs?  Is it by observed behavior, or just recommended by  
other hadoop users?

- Use a good server with lots (15GB+) of RAM.
      - why 15+ GBs?  Do we allocate all memory to the NameNode? or  
just allocate some number using -Xmx and leave the rest available so  
the machine doesnt start swapping?

- Consider using fast RAID5 storage for keeping the index.
      - why RAID5?

- List more than one name node directory in the configuration, so  
that multiple copies of the indices will be stored. As long as the  
directories are on separate disks, a single full disk will not  
corrupt the index.
      - If running RAID 5, why is this necessary?

- Configure the name node to store one set of transaction logs on a  
separate disk from the index.
      - why?

- Configure the name node to store another set of transaction logs to  
a network mounted disk.
      - why?

- Do not host DataNode, JobTracker or TaskTracker services on the  
same system.
      - how much memory would the job tracker need?  Does it use a  
lot of CPU? In general, what are good specs for a job tracker machine  
and can the machine be shared with other services?

Thanks so much for the help.  I think it would be hugely helpful for  
the community to start describing their respective setups for hadoop  
clusters in more detail than just the config for datanodes and  
cluster size.  I think we all want to be confident that we are  
spending money on the right machines to grow our cluster the right way.

Most appreciated,

- Manish
Co-Founder Rapleaf.com

We're looking for a product manager, sys admin, and software  
engineers...$10K referral award

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message