Mailing-List: contact core-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-user@hadoop.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; s=serpent; d=yahoo-inc.com; c=nofws; q=dns;
	h=received:x-mimeole:content-class:mime-version:
	content-type:content-transfer-encoding:subject:date:message-id:
	in-reply-to:x-ms-has-attach:x-ms-tnef-correlator:thread-topic:
	thread-index:references:from:to:return-path:x-originalarrivaltime;
	b=2lX+RuIJ7yGg+KgC91UEeEvInYXIsuhr+pzBb2fSPBQMxe8CcHrW1DYFySYMmzcW
Content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Subject: RE: long write operations and data recovery
Date: Thu, 28 Feb 2008 21:53:59 -0800
Message-ID: 
 <84E52AD05F6F884AAFF3344FE4C95991011E5DB4@SNV-EXVS08.ds.corp.yahoo.com>
In-Reply-To: 
 <FCDAD498A048A6428B2255D5D4587D3702675901@sf2pmxb01.thefacebook.com>
Thread-Topic: long write operations and data recovery
Thread-Index: Ach6bdEUYVT+s7naT6yc45AquJDsngABy2igAAhvs3A=
References: <47C757C5.5040403@invitemedia.com>
 <FCDAD498A048A6428B2255D5D4587D3702675901@sf2pmxb01.thefacebook.com>
From: "dhruba Borthakur" <dhruba@yahoo-inc.com>
To: <core-user@hadoop.apache.org>

I agree with Joydeep. For batch processing, it is sufficient to make the
application not assume that HDFS is always up and active. However, for
real-time applications that are not batch-centric, it might not be
sufficient. There are a few things that HDFS could do to better handle
Namenode outages:

1. Make Clients handle transient Namenode downtime. This requires that
Namenode restarts are fast, clients can handle long Namenode outages,
etc.etc.
2. Design HDFS Namenode to be a set of two, an active one and a passive
one. The active Namenode could continuously forward transactions to the
passive one. In case of failure of the active Namenode, the passive
could take over. This type of High-Availability would probably be very
necessary for non-batch-type-applications.

Thanks,
dhruba

-----Orivery necessaginal Message-----
From: Joydeep Sen Sarma [mailto:jssarma@facebook.com]=20
Sent: Thursday, February 28, 2008 6:06 PM
To: core-user@hadoop.apache.org
Subject: RE: long write operations and data recovery

We have had a lot of peace of mind by building a data pipeline that does
not assume that hdfs is always up and running. If the application is
primarily non real-time log processing - I would suggest
batch/incremental copies of data to hdfs that can catch up automatically
in case of failures/downtimes.

we have a rsync like map-reduce job that monitors a log directories and
keeps pulling new data in (and suspect lot of other users do similar
stuff as well). Might be a useful notion to generalize and put in
contrib.


-----Original Message-----
From: Steve Sapovits [mailto:ssapovits@invitemedia.com]=20
Sent: Thursday, February 28, 2008 4:54 PM
To: core-user@hadoop.apache.org
Subject: Re: long write operations and data recovery


> How does replication affect this?  If there's at least one replicated
>  client still running, I assume that takes care of it?

Never mind -- I get this now after reading the docs again.

My remaining point of failure question concerns name nodes.  The docs
say manual=20
intervention is still required if a name node goes down.  How is this
typically managed
in production environments?   It would seem even a short name node
outage in a=20
data intestive environment would lead to data loss (no name node to give
the data
to).

--=20
Steve Sapovits
Invite Media  -  http://www.invitemedia.com
ssapovits@invitemedia.com