Mailing-List: contact hdfs-user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-user@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of sean.bigdatafun@gmail.com
 designates 209.85.210.176 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :cc:content-type;
        b=YUHgApXQ1JxWWDspXXnPdPEudb7STb2XSDJnSvNy2FvTAQlaYViGz9en4XXMmihf/I
         IMzhmxMLp+KPqLJyU7jhbRD3wn/e/pZltINu7Rb83VzhQ+wqXeQuz8R7Vh4F+bPbkYnw
         bccrwxif/HAgfun2x2qzdhsvwqS59f9+fyQCE=
MIME-Version: 1.0
In-Reply-To: <AANLkTik3wmnaMtWfVEQK8cgasnUvCTBvLTVaSrhmNhs9@mail.gmail.com>
References: <AANLkTimGDTHAnQ47-nHTimENvT64f_JMhp7a8gC1dDyr@mail.gmail.com>
	<AANLkTik3wmnaMtWfVEQK8cgasnUvCTBvLTVaSrhmNhs9@mail.gmail.com>
Date: Sun, 13 Mar 2011 22:09:20 -0700
Message-ID: <AANLkTi=NMGxsZqC_rDD6wWW3ok1bsxpkSYphvBNVxDh8@mail.gmail.com>
Subject: Re: Will blocks of an unclosed file get lost when HDFS client (or the
 HDFS cluster) crashes?
From: Sean Bigdatafun <sean.bigdatafun@gmail.com>
To: Ted Dunning <tdunning@maprtech.com>
Cc: hdfs-user@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1

I meant an HDFS chunk (the size of 64MB), and I meant the version of
0.20.2 without append patch.

I think even without the append patch, the previous 64MB blocks (in my
example, the first 5 blocks) should be safe. Isn't it?


On 3/13/11, Ted Dunning <tdunning@maprtech.com> wrote:
> What do you mean by block?  An HDFS chunk?  Or a flushed write?
>
> The answer depends a bit on which version of HDFS / Hadoop you are using.
>  With the append branches, things happen a lot more like what you expect.
>  Without that version, it is difficult to say what will happen.
>
> Also, there are very few guarantees about what happens if the namenode
> crashes.  There are some provisions for recovery, but none of them really
> have any sort of transactional guarantees.  This means that there may be
> some unspecified time before the writes that you have done are actually
> persisted in a recoverable way.
>
> On Sun, Mar 13, 2011 at 9:52 AM, Sean Bigdatafun
> <sean.bigdatafun@gmail.com>wrote:
>
>> Let's say an HDFS client starts writing a file A (which is 10 blocks
>> long) and 5 blocks have been writen to datanodes.
>>
>> At this time, if the HDFS client crashes (apparently without a close
>> op), will we see 5 valid blocks for file A?
>>
>> Similary, at this time if the HDFS cluster crashes, will we see 5
>> valid blocks for file A?
>>
>> (I guess both answers are yes, but I'd have some confirmation :-)
>> --
>> --Sean
>>
>


-- 
--Sean