hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael Segel <michael_se...@hotmail.com>
Subject RE: what will happen if a backup name node folder becomes unaccessible?
Date Mon, 23 Aug 2010 19:05:05 GMT

Ok... 

Now you have me confused.
Everything we've seen says that writing to both a local disk and to an NFS mounted disk would
be the best way to prevent a problem.

Now you and Harsh J say that this could actually be problematic. 

Which is it?
Is this now a defect that should be addressed, or should we just not use an NFS mounted drive?

Thx

-Mike


> Date: Mon, 23 Aug 2010 11:42:59 -0700
> From: licht_jiang@yahoo.com
> Subject: Re: what will happen if a backup name node folder becomes unaccessible?
> To: common-user@hadoop.apache.org
> 
> This makes a good argument. Actually, after seeing the previous reply, I kindof convinced
that I should go back to "sync" the meta data to a backup location instead of using this feature,
which as David mentioned, introduced a 2nd single point of failure to hadoop, which degrades
the availability of hadoop. BTW, we are using cloudera package hadoop-0.20.2+228. Can someone
confirm whether a name node will shut down given that a backup folder listed in "dfs.name.dir"
becomes unavailable in this version?
> 
> Thanks,
> 
> Michael
> 
> --- On Sun, 8/22/10, David B. Ritch <david.ritch@gmail.com> wrote:
> 
> From: David B. Ritch <david.ritch@gmail.com>
> Subject: Re: what will happen if a backup name node folder becomes unaccessible?
> To: common-user@hadoop.apache.org
> Date: Sunday, August 22, 2010, 11:34 PM
> 
>  Which version of Hadoop was this?  The folks at Cloudera have assured
> me that the namenode in CDH2 will continue as long as one of the
> directories is still writable.
> 
> It *does* seem a bit of a waste if an availability feature - the ability
> to write to multiple directories - actually reduces availability by
> providing an additional single point of failure.
> 
> Thanks!
> 
> dbr
> 
> On 8/20/2010 5:27 PM, Harsh J wrote:
> > Whee, lets try it out:
> >
> > Start with both paths available. ... Starts fine.
> > Store some files. ... Works.
> > rm -r the second path. ... Ouch.
> > Store some more files. ... Still Works. [Cuz the SNN hasn't sent us
> > stuff back yet]
> > Wait for checkpoint to hit.
> > And ...
> > Boom!
> >
> > 2010-08-21 02:42:00,385 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Roll Edit Log
> > from 127.0.0.1
> > 2010-08-21 02:42:00,385 INFO
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of
> > transactions: 37 Total time for transactions(ms): 6Number of
> > transactions batched in Syncs: 0 Number of syncs: 26 SyncTimes(ms):
> > 307 277
> > 2010-08-21 02:42:00,439 FATAL
> > org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Fatal Error : All
> > storage directories are inaccessible.
> > 2010-08-21 02:42:00,440 INFO
> > org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
> > /************************************************************
> > SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
> > ************************************************************/
> >
> > So yes, as Edward says - never let this happen!
> >
> > On Sat, Aug 21, 2010 at 2:26 AM, jiang licht <licht_jiang@yahoo.com> wrote:
> >> Using nfs folder to back up dfs meta information as follows,
> >>
> >> <property>
> >>         <name>dfs.name.dir</name>
> >>         <value>/hadoop/dfs/name,/hadoop-backup/dfs/name</value>
> >>     </property>
> >>
> >> where /hadoop-backup is on a backup machine and mounted on the master node.
> >>
> >> I have a question: if somehow, the backup folder becomes unavailable, will it
freeze master node? That is, will write operation simply hang up on this condition on the
master node? Or will master node log the problem and continues to work?
> >>
> >> Thanks,
> >>
> >> Michael
> >>
> >>
> >>
> >
> >
> 
> 
> 
> 
>       
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message