hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sudhir Vallamkondu <Sudhir.Vallamko...@icrossing.com>
Subject Re: common-user Digest 23 Aug 2010 21:21:26 -0000 Issue 1518
Date Tue, 24 Aug 2010 05:19:14 GMT
Looking at the codebase it seems to suggest that it ignores a editlog
storage directory if it encounters an error

http://www.google.com/codesearch/p?hl=en#GLh8vwsjDqs/trunk/src/hdfs/org/apac
he/hadoop/hdfs/server/namenode/FSEditLog.java&q=namenode%20editlog&sa=N&cd=2
0&ct=rc

Check lines:
Code in line 334
comment: 387 - 390
comment: 411 - 414
Comment: 433 - 436

The processIOError method is called throughout the code if it encounters an
IOException.  

A fatal error is only thrown if none of the storage directories is
accessible. Lines 394, 420

- Sudhir



On Aug/23/ 2:21 PM, "common-user-digest-help@hadoop.apache.org"
<common-user-digest-help@hadoop.apache.org> wrote:

> From: Michael Segel <michael_segel@hotmail.com>
> Date: Mon, 23 Aug 2010 14:05:05 -0500
> To: <common-user@hadoop.apache.org>
> Subject: RE: what will happen if a backup name node folder becomes
> unaccessible?
> 
> 
> Ok... 
> 
> Now you have me confused.
> Everything we've seen says that writing to both a local disk and to an NFS
> mounted disk would be the best way to prevent a problem.
> 
> Now you and Harsh J say that this could actually be problematic.
> 
> Which is it?
> Is this now a defect that should be addressed, or should we just not use an
> NFS mounted drive?
> 
> Thx
> 
> -Mike
> 
> 
>> Date: Mon, 23 Aug 2010 11:42:59 -0700
>> From: licht_jiang@yahoo.com
>> Subject: Re: what will happen if a backup name node folder becomes
>> unaccessible?
>> To: common-user@hadoop.apache.org
>> 
>> This makes a good argument. Actually, after seeing the previous reply, I
>> kindof convinced that I should go back to "sync" the meta data to a backup
>> location instead of using this feature, which as David mentioned, introduced
>> a 2nd single point of failure to hadoop, which degrades the availability of
>> hadoop. BTW, we are using cloudera package hadoop-0.20.2+228. Can someone
>> confirm whether a name node will shut down given that a backup folder listed
>> in "dfs.name.dir" becomes unavailable in this version?
>> 
>> Thanks,
>> 
>> Michael
>> 
>> --- On Sun, 8/22/10, David B. Ritch <david.ritch@gmail.com> wrote:
>> 
>> From: David B. Ritch <david.ritch@gmail.com>
>> Subject: Re: what will happen if a backup name node folder becomes
>> unaccessible?
>> To: common-user@hadoop.apache.org
>> Date: Sunday, August 22, 2010, 11:34 PM
>> 
>>  Which version of Hadoop was this?  The folks at Cloudera have assured
>> me that the namenode in CDH2 will continue as long as one of the
>> directories is still writable.
>> 
>> It *does* seem a bit of a waste if an availability feature - the ability
>> to write to multiple directories - actually reduces availability by
>> providing an additional single point of failure.
>> 
>> Thanks!
>> 
>> dbr
>> 
>> On 8/20/2010 5:27 PM, Harsh J wrote:
>>> Whee, lets try it out:
>>> 
>>> Start with both paths available. ... Starts fine.
>>> Store some files. ... Works.
>>> rm -r the second path. ... Ouch.
>>> Store some more files. ... Still Works. [Cuz the SNN hasn't sent us
>>> stuff back yet]
>>> Wait for checkpoint to hit.
>>> And ...
>>> Boom!
>>> 
>>> 2010-08-21 02:42:00,385 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
>>> from 127.0.0.1
>>> 2010-08-21 02:42:00,385 INFO
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
>>> transactions: 37 Total time for transactions(ms): 6Number of
>>> transactions batched in Syncs: 0 Number of syncs: 26 SyncTimes(ms):
>>> 307 277
>>> 2010-08-21 02:42:00,439 FATAL
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
>>> storage directories are inaccessible.
>>> 2010-08-21 02:42:00,440 INFO
>>> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>>> /************************************************************
>>> SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
>>> ************************************************************/
>>> 
>>> So yes, as Edward says - never let this happen!
>>> 
>>> On Sat, Aug 21, 2010 at 2:26 AM, jiang licht <licht_jiang@yahoo.com> wrote:
>>>> Using nfs folder to back up dfs meta information as follows,
>>>> 
>>>> <property>
>>>>         <name>dfs.name.dir</name>
>>>>         <value>/hadoop/dfs/name,/hadoop-backup/dfs/name</value>
>>>>     </property>
>>>> 
>>>> where /hadoop-backup is on a backup machine and mounted on the master node.
>>>> 
>>>> I have a question: if somehow, the backup folder becomes unavailable, will
>>>> it freeze master node? That is, will write operation simply hang up on this
>>>> condition on the master node? Or will master node log the problem and
>>>> continues to work?
>>>> 
>>>> Thanks,
>>>> 
>>>> Michael
>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
>> 
>> 
>>       


iCrossing Privileged and Confidential Information
This email message is for the sole use of the intended recipient(s) and may contain confidential
and privileged information of iCrossing. Any unauthorized review, use, disclosure or distribution
is prohibited. If you are not the intended recipient, please contact the sender by reply email
and destroy all copies of the original message.



Mime
View raw message