hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Colin McCabe <cmcc...@alumni.cmu.edu>
Subject Re: Using a hard drive instead of
Date Wed, 17 Oct 2012 23:21:01 GMT
Hi Mark,

HDFS contains a write-ahead log which will protect you from power
failure.  It's called the edit log.  If you want warm failover, you
can use HDFS HA, which is available in recent versions of HDFS.

Hope this helps.

On Wed, Oct 17, 2012 at 3:44 PM, Mark Kerzner <mark.kerzner@shmsoft.com> wrote:
> Colin,
> swap space would give me very high memory, up to 1 TB, say, but it won't
> protect me from power failure. Very large clusters is only one application
> of this idea, warm failover is another. The drive is not theoretical, just
> look at the price tag :)
> Mark
> On Wed, Oct 17, 2012 at 5:37 PM, Colin Patrick McCabe <rarecactus@gmail.com>
> wrote:
>> The direct answer to your question us to use this theoretical super-fast
>> hard drive as Linux swap space.
>> The better answer is to use federation or another solution if your needs
>> exceed those servable by a single NameNode.
>> Cheers.
>> Colin
>> On Oct 11, 2012 9:00 PM, "Mark Kerzner" <mark.kerzner@shmsoft.com> wrote:
>>> Hi,
>>> Imagine I have a very fast hard drive that I want to use for the
>>> NameNode. That is, I want the NameNode to store its blocks information on
>>> this hard drive instead of in memory.
>>> Why would I do it? Scalability (no federation needed), many files are not
>>> a problem, and warm fail-over is automatic. What would I need to change in
>>> the NameNode to tell it to use the hard drive?
>>> Thank you,
>>> Mark

View raw message