hadoop-general mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From gmac...@cs.ucf.edu
Subject Re: Namenode with External Storage?
Date Thu, 22 Oct 2009 16:37:21 GMT
As with Dhruba's comment, so long as it is just the namenode that is 
running on a networked file system everything should be chill. The namenode 
keeps all of its working metadata in main mem, and it only occasionally 
pushes a log file out to hard storage (and if I remember correctly you can 
adjust this time window in one of the site files).

However, you are going to run into huge performance issues running 
datanodes over a networked storage system. Having to push that many file 
requests over a network for a respectable mapreduce job is going to kill 
your equipment.

 - Grant

On Oct 21 2009, Jonathan Seidman wrote:

>Apologies if this has been answered previously, but I'm unable to find
>anything that seems to cover this.
> It's clear that datanodes require local storage for Hadoop to function 
> efficiently, but is there any significant disadvantage to using external 
> storage for namenodes? We're exploring the possibility of using a 
> different class of hardware for our namenodes with attached storage and 
> little or no internal storage. Some of the benefits this would provide us 
> are: 1) allowing our sysadmins to deploy hardware that they're familiar 
> with and already have considerable experience keeping up in a production 
> environment. 2) no namenode downtime to replace a failed disk.
>We don't anticipate that this approach would cause any significant
>degradation to performance, but let me know if there's something we're not

Grant Mackey
PhD student Computer Engineering
University of Central Florida
Rm 231 cube 5 (321) 960-8851

View raw message