hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Baldeschwieler <eri...@yahoo-inc.com>
Subject Re: Hadoop 0.20.0
Date Thu, 26 Feb 2009 04:45:21 GMT

Hi Folks,

I don't think vetoing releases until the append issue is resolved is  
constructive.  I'd like to invite folks who want to fix append to work  
with us on that and support releasing 20 without it in parallel.

Please keep in mind the spirit of this project.  Putting your shoulder  
to the wheel gets features added to the project.  Vetoing a release  
will not get anything added to it.  It will simply delay getting those  
features we've all agreed work tested and released.  Resolving the  
append issues is going to take a big investment.  We all want to see  
that happen.  But we also need to provide the whole hadoop community  
with stable releases while we do the hard work needed to really solve  
this problem.

Why are we taking this position?  The current append design has proven  
fix resistant.  We've deployed fix after fix and have continued to see  
new append bugs that have caused data loss and cluster down time.  We  
moved to disable append in our clusters because that was the only path  
we could see to delivering a stable system to our users.  We believe  
the only way to solve this problem is to acknowledge that the append  
code is a critical section of the code that needs to be designed,  
documented and tested thoroughly if we expect it to function well.   
Given the pain the append issue has caused us, we are going to -1  
"quick fixes" to it.  Data loss and corruption are serious issues.   
We've been burned.

I hope folks will consider this and both support a 20 release without  
append and also work with us to really fix append.  We promise to work  
hard with the community to deliver a 20.x or 21 release with append.


eric14 a.k.a. Eric Baldeschwieler
vp grid computing
Yahoo!  Inc.

On Feb 25, 2009, at 7:34 PM, Nigel Daley wrote:

> On Feb 24, 2009, at 9:28 PM, Dhruba Borthakur wrote:
>> Hi Jim,
>> I can understand your problem.  I can probably whip out a fix for
>> HADOOP-4663 and HADOOP-4379 by the end of this week. It would be  
>> nice if
>> somebody else (Hairong,  Sanjay, Konstantin?) can volunteer to  
>> discuss and
>> review the patches/fixes.
> "Whipping out a patch" doesn't give me any confidence that this  
> feature will be fixed properly.  We're building a file system.  Data  
> reliability and accuracy are absolutely key.  We know that this  
> feature has been very lightly tested.
>> Nigel: wht is the proposed deadline for 0.20?
> March 6.
> Nige
>> thanks,
>> dhruba
>> On Tue, Feb 24, 2009 at 4:25 PM, Jim Kellerman (POWERSET) <
>> Jim.Kellerman@microsoft.com> wrote:
>>> --1
>>> HBase really needs 4379. My testing to date indicates that it does  
>>> work
>>> (although I have a bit more testing to do).
>>> I was ok with not putting it into 0.19.1 provided it was in 0.19.2  
>>> and
>>> 0.20.0.
>>> It's a big problem for us now and is hurting our ability to keep our
>>> community alive. (They will go to Cassandra or something else to  
>>> ensure
>>> reliability).
>>>> -----Original Message-----
>>>> From: Nigel Daley [mailto:ndaley@yahoo-inc.com]
>>>> Sent: Tuesday, February 24, 2009 4:02 PM
>>>> To: core-dev@hadoop.apache.org
>>>> Subject: Hadoop 0.20.0
>>>> Folks,
>>>> Hadoop 0.19.1 is now available with the file append feature  
>>>> disabled.
>>>> It's time to talk about a Hadoop 0.20.0 release.
>>>> Hadoop 0.20.0 feature freeze date was almost 3 months ago.  The  
>>>> last
>>>> few blockers are now almost fixed (should be next week) except for
>>>> HADOOP-4379.  HADOOP-4379 is work that is needed to properly  
>>>> implement
>>>> file append.
>>>> *** I propose we move HADOOP-4379 off to release 0.21.0 and apply  
>>>> the
>>>> same disabling of file append in Hadoop 0.20.0 that we put in  
>>>> place to
>>>> get 0.19.1 released (HADOOP-5224 and HADOOP-5225).
>>>> I will call a vote for 0.20.0 when blockers are fixed.
>>>> Cheers,
>>>> Nigel
>>>>> Folks,
>>>>> Some Hadoop deployments have upgraded to 0.19.0.  Clearly, the  
>>>>> 0.19
>>>>> branch has issues and a 0.19.1 release is needed.
>>>>> Quality issues in the changes made for the file append feature  
>>>>> have
>>>>> prevented some from deploying Hadoop 0.19.  One of these changes
>>>>> (sync) has now been "fixed" by reducing its semantics in Hadoop
>>>>> 0.18.3 (HADOOP-4997).  This was necessary to stabilize the 0.18
>>>>> branch.
>>>>> I would like to propose that we apply this same "fix" to sync in
>>>>> 0.19.1 and 0.20.0.  Since append requires the full semantics of
>>>>> sync, I propose we also disable append (perhaps throw
>>>>> UnsupportedOperationException from API?).  Yes, this would
>>>>> unfortunately be an incompatible change between 0.19.0 and 0.19.1.
>>>>> We can then take the time needed to fix append properly in 0.21.0.
>>>>> I will call a vote for 0.19.1 and 0.20.0 when blockers are fixed.
>>>>> Nigel

View raw message