Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of rarecactus@gmail.com designates
 209.85.212.176 as permitted sender)
MIME-Version: 1.0
Sender: rarecactus@gmail.com
In-Reply-To: <CB928283.8DE9F%scott@richrelevance.com>
References: 
 <CAPn_vTuBTU=TN+MocUwgxbDs2pGrkYnD17E+F0LFyfJd5QPGeQ@mail.gmail.com>
	<CB928283.8DE9F%scott@richrelevance.com>
Date: Mon, 26 Mar 2012 12:53:59 -0700
Message-ID: 
 <CA+qbEUPJN8buvcbAs1smX7vLE+r=g=7Xy98gbyXRADuwHf56vA@mail.gmail.com>
Subject: Re: [DISCUSS] Remove append?
From: Colin McCabe <cmccabe@alumni.cmu.edu>
To: hdfs-dev@hadoop.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Fri, Mar 23, 2012 at 7:44 PM, Scott Carey <scott@richrelevance.com> wrot=
e:
>
>
> On 3/22/12 10:25 AM, "Eli Collins" <eli@cloudera.com> wrote:
>
>>On Thu, Mar 22, 2012 at 1:26 AM, Konstantin Shvachko
>><shv.hadoop@gmail.com> wrote:
>>> Eli,
>>>
>>> I went over the entire discussion on the topic, and did not get it. Is
>>> there a problem with append? We know it does not work in hadoop-1,
>>> only flush() does. Is there anything wrong with the new append
>>> (HDFS-265)? If so please file a bug.
>>> I tested it in Hadoop-0.22 branch it works fine.
>>>
>>> I agree with people who were involved with the implementation of the
>>> new append that the complexity is mainly in
>>> 1. pipeline recovery
>>> 2. consistent client reading while writing, and
>>> 3. hflush()
>>> Once it is done the append itself, which is reopening of previously
>>> closed files for adding data, is not complex.
>>>
>>
>>I agree that much of the complexity is in #1-3 above, which is why
>>HDFS-265 is leveraged.
>>The primary simplicity of not having append (and truncate) comes from
>>not leveraging the invariant that finalized blocks are immutable, that
>>blocks once written won't eg shrink in size (which we assume today).
>
> That invariant can co-exist with append via copy-on-write. =A0The new sta=
te
> and old state would co-exist until the old state was not needed, a file's
> block map would have to use a persistent data structure. Copy on write
> semantics with blocks in file systems is all the rage these days. =A0Free
> snapshots, atomic transactions for operations on multiple blocks, etc.

Hi Scott,

If a client accesses a file, and then the client becomes unresponsive,
how long should you wait before declaring the blocks he was looking at
unused?  No matter how long or how short a period you choose, someone
will argue with it.  And having to track this kind of state in the
NameNode introduces a huge amount of complexity, not to mention extra
memory consumption.  Basically, we would have to track the ID of every
block that any client looked at, at all times.

Colin


>
>>
>>> You mentioned it and I agree you indeed should be more involved with
>>> your customer base. As for eBay, append was of the motivations to work
>>> on stabilizing 0.22 branch. And there is a lot of use cases which
>>> require append for our customers.
>>> Some of them were mentioned in this discussion.
>>>
>>
> >From what I've seen 0.22 isn't ready for production use. Aside from
>>not supporting critical features like security, it doesn't have a
>>size-able user-base behind it testing and fixing bugs, etc. All things
>>I'd imagine an org like eBay would want. =A0I've never gotten a request
>>to support 0.22 from a customer.
>>
>>Thanks,
>>Eli
>