hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: HDFS Backup nodes
Date Wed, 14 Dec 2011 02:19:01 GMT
On Sun, Dec 11, 2011 at 10:47 PM, M. C. Srivas <mcsrivas@gmail.com> wrote:
> But if you use a Netapp, then the likelihood of the Netapp crashing is
> lower than the likelihood of a garbage-collection-of-death happening in the
> NN.

This is pure FUD.

I've never seen a "garbage collection of death" ever in any NN with
smaller than a 40GB heap, and only a small handful of times on larger
heaps. So, unless you're running a 4000 node cluster, you shouldn't be
concerned with this. And the existence of many 4000 node clusters
running fine on HDFS indicates that a properly tuned NN does just
fine.

[Disclaimer: I don't spread FUD regardless of vendor affiliation.]

-Todd

>
> [ disclaimer:  I don't work for Netapp, I work for MapR ]
>
>
> On Wed, Dec 7, 2011 at 4:30 PM, randy <randysch@comcast.net> wrote:
>
>> Thanks Joey. We've had enough problems with nfs (mainly under very high
>> load) that we thought it might be riskier to use it for the NN.
>>
>> randy
>>
>>
>> On 12/07/2011 06:46 PM, Joey Echeverria wrote:
>>
>>> Hey Rand,
>>>
>>> It will mark that storage directory as failed and ignore it from then
>>> on. In order to do this correctly, you need a couple of options
>>> enabled on the NFS mount to make sure that it doesn't retry
>>> infinitely. I usually run with the tcp,soft,intr,timeo=10,**retrans=10
>>> options set.
>>>
>>> -Joey
>>>
>>> On Wed, Dec 7, 2011 at 12:37 PM,<randysch@comcast.net>  wrote:
>>>
>>>> What happens then if the nfs server fails or isn't reachable? Does hdfs
>>>> lock up? Does it gracefully ignore the nfs copy?
>>>>
>>>> Thanks,
>>>> randy
>>>>
>>>> ----- Original Message -----
>>>> From: "Joey Echeverria"<joey@cloudera.com>
>>>> To: common-user@hadoop.apache.org
>>>> Sent: Wednesday, December 7, 2011 6:07:58 AM
>>>> Subject: Re: HDFS Backup nodes
>>>>
>>>> You should also configure the Namenode to use an NFS mount for one of
>>>> it's storage directories. That will give the most up-to-date back of
>>>> the metadata in case of total node failure.
>>>>
>>>> -Joey
>>>>
>>>> On Wed, Dec 7, 2011 at 3:17 AM, praveenesh kumar<praveenesh@gmail.com>
>>>>  wrote:
>>>>
>>>>> This means still we are relying on Secondary NameNode idealogy for
>>>>> Namenode's backup.
>>>>> Can OS-mirroring of Namenode is a good alternative keep it alive all
the
>>>>> time ?
>>>>>
>>>>> Thanks,
>>>>> Praveenesh
>>>>>
>>>>> On Wed, Dec 7, 2011 at 1:35 PM, Uma Maheswara Rao G<
>>>>> maheswara@huawei.com>wrote:
>>>>>
>>>>>  AFAIK backup node introduced in 0.21 version onwards.
>>>>>> ______________________________**__________
>>>>>> From: praveenesh kumar [praveenesh@gmail.com]
>>>>>> Sent: Wednesday, December 07, 2011 12:40 PM
>>>>>> To: common-user@hadoop.apache.org
>>>>>> Subject: HDFS Backup nodes
>>>>>>
>>>>>> Does hadoop 0.20.205 supports configuring HDFS backup nodes ?
>>>>>>
>>>>>> Thanks,
>>>>>> Praveenesh
>>>>>>
>>>>>>
>>>>
>>>>
>>>> --
>>>> Joseph Echeverria
>>>> Cloudera, Inc.
>>>> 443.305.9434
>>>>
>>>
>>>
>>>
>>>
>>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
View raw message