hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Changes to FSImage/FSEditLog
Date Wed, 24 Nov 2010 22:25:02 GMT
Hi all,

There's a patch up for 1473 now. If we can get a +1 we can commit this and
move on to the next steps in some of these projects.

Thanks
-Todd

On Thu, Nov 18, 2010 at 12:42 AM, Ivan Kelly <ivank@yahoo-inc.com> wrote:

> This patch looks really good. It shouldn't affect our refactoring too much
> and will actually make things a lot easier for the next part (breaking the
> circular dependency between namesystem and fsimage).
>
> -Ivan
>
>
> On 18 Nov 2010, at 02:27, Todd Lipcon wrote:
>
> I put up a sketch of a patch for 1473. It compiles and passes at least a
> couple of the unit tests, but more work to be done on it. Would appreciate
> comments from this crowd if the new classes seem like good divisions.
>
> -Todd
>
> On Wed, Nov 17, 2010 at 2:36 PM, Todd Lipcon <todd@cloudera.com> wrote:
>
>> On Wed, Nov 17, 2010 at 2:34 PM, Hairong Kuang <kuang.hairong@fb.com>wrote:
>>
>>> You meant HDFS-1473 right? When do you expect that it could be done?
>>>
>>>
>> I was working on HDFS-1473 this morning and could probably get a
>> preliminary patch up today.. but when I realized that we have 4 parallel
>> refactorings going on I paused my work. If we agree that HDFS-1473 will go
>> in before the other ones I can finish it up, but if we're doing the same
>> thing we should coordinate instead.
>>
>>
>>> Hairong
>>>
>>> On 11/17/10 2:29 PM, "Eli Collins" <eli@cloudera.com> wrote:
>>>
>>> > Agree we should do the refactoring before the features, we should also
>>> > add the tests with the refactoring to show they don't break things.
>>> >
>>> > On Wed, Nov 17, 2010 at 2:27 PM, Todd Lipcon <todd@cloudera.com>
>>> wrote:
>>> >> OK, looks like there is still quite a bit more in motion here, thanks
>>> for
>>> >> the list, Hairong.
>>> >> How would everyone feel about setting aside those improvements/fixes,
>>> doing
>>> >> the refactor first, and then going back to them? It should help with
>>> testing
>>> >> and reduce conflicts between the various ongoing projects.
>>> >> Maybe we can discuss quickly on this thread: what are some small
>>> refactors
>>> >> we can start with to help separate these things? Ivan's project looks
>>> really
>>> >> good but it seems like it is a pretty large change. If we can break
it
>>> up
>>> >> into a couple smaller patches we can commit as we go.
>>> >> -Todd
>>> >> On Wed, Nov 17, 2010 at 2:08 PM, Hairong Kuang <kuang.hairong@fb.com>
>>> wrote:
>>> >>>
>>> >>> Hi Todd,
>>> >>>
>>> >>> Here is the list of fsimage jiras that I am working on:
>>> >>>
>>> >>> HDFS-1481 // small change
>>> >>> HDFS-1458 // small change
>>> >>> HDFS-1496  // should be checked into 022, I still have no clue how
to
>>> fix
>>> >>> it
>>> >>> HDFS-1070  // this one has a simple idea, but turns out that it
needs
>>> >>> major code re-organization. I am still struggling with how to make
my
>>> code
>>> >>> change easier to review. But this will definitely collide with yours.
>>> >>>
>>> >>> Hairong
>>> >>>
>>> >>> On 11/17/10 11:57 AM, "Todd Lipcon" <todd@cloudera.com> wrote:
>>> >>>
>>> >>> Hey all,
>>> >>>
>>> >>> As many of you know, I've been working on HDFS-1073 for a few months,
>>> and
>>> >>> we're hoping to get it in for 0.22 (OOM has given the OK to merge
>>> this into
>>> >>> branch and Sanjay has helpfully gotten some commitment from Yahoo
QA
>>> to help
>>> >>> test it).
>>> >>>
>>> >>> It's been difficult recently to work on the patch as a lot of changes
>>> have
>>> >>> gone in around FSImage and FSEditLog (eg checksumming, parallel
load,
>>> etc)
>>> >>> so the patch falls out of date quickly and is hard to resolve the
>>> changes
>>> >>> since my patch moves a lot of code. Ivan Kelly over at Yahoo
>>> Barcelona has
>>> >>> also been working on a refactor and I think running into a similar
>>> issue.
>>> >>>
>>> >>> So, I just wanted to ping everyone who has been working on this
area
>>> of
>>> >>> the code to find out if there are any more remaining JIRAs that
>>> you're
>>> >>> hoping to commit in the next several weeks that will touch FSImage
>>> and
>>> >>> FSEditLog. If so, we should try to agree on an order in which to
>>> commit them
>>> >>> so we don't end up stomping on each other, and we can help review
>>> each
>>> >>> other's work to move quicker.
>>> >>>
>>> >>> If you can just reply with a list of any remaining uncommitted JIRAs
>>> that
>>> >>> touch FSImage/FSEditLog that would be great, then we can work from
>>> there on
>>> >>> how to proceed and minimize the amount of patch rebasing we all
have
>>> to do.
>>> >>>
>>> >>> Thanks!
>>> >>> -Todd
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Todd Lipcon
>>> >> Software Engineer, Cloudera
>>> >>
>>>
>>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>
>
>


-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message