hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: Changes to FSImage/FSEditLog
Date Thu, 18 Nov 2010 01:27:01 GMT
I put up a sketch of a patch for 1473. It compiles and passes at least a
couple of the unit tests, but more work to be done on it. Would appreciate
comments from this crowd if the new classes seem like good divisions.

-Todd

On Wed, Nov 17, 2010 at 2:36 PM, Todd Lipcon <todd@cloudera.com> wrote:

> On Wed, Nov 17, 2010 at 2:34 PM, Hairong Kuang <kuang.hairong@fb.com>wrote:
>
>> You meant HDFS-1473 right? When do you expect that it could be done?
>>
>>
> I was working on HDFS-1473 this morning and could probably get a
> preliminary patch up today.. but when I realized that we have 4 parallel
> refactorings going on I paused my work. If we agree that HDFS-1473 will go
> in before the other ones I can finish it up, but if we're doing the same
> thing we should coordinate instead.
>
>
>> Hairong
>>
>> On 11/17/10 2:29 PM, "Eli Collins" <eli@cloudera.com> wrote:
>>
>> > Agree we should do the refactoring before the features, we should also
>> > add the tests with the refactoring to show they don't break things.
>> >
>> > On Wed, Nov 17, 2010 at 2:27 PM, Todd Lipcon <todd@cloudera.com> wrote:
>> >> OK, looks like there is still quite a bit more in motion here, thanks
>> for
>> >> the list, Hairong.
>> >> How would everyone feel about setting aside those improvements/fixes,
>> doing
>> >> the refactor first, and then going back to them? It should help with
>> testing
>> >> and reduce conflicts between the various ongoing projects.
>> >> Maybe we can discuss quickly on this thread: what are some small
>> refactors
>> >> we can start with to help separate these things? Ivan's project looks
>> really
>> >> good but it seems like it is a pretty large change. If we can break it
>> up
>> >> into a couple smaller patches we can commit as we go.
>> >> -Todd
>> >> On Wed, Nov 17, 2010 at 2:08 PM, Hairong Kuang <kuang.hairong@fb.com>
>> wrote:
>> >>>
>> >>> Hi Todd,
>> >>>
>> >>> Here is the list of fsimage jiras that I am working on:
>> >>>
>> >>> HDFS-1481 // small change
>> >>> HDFS-1458 // small change
>> >>> HDFS-1496  // should be checked into 022, I still have no clue how to
>> fix
>> >>> it
>> >>> HDFS-1070  // this one has a simple idea, but turns out that it needs
>> >>> major code re-organization. I am still struggling with how to make my
>> code
>> >>> change easier to review. But this will definitely collide with yours.
>> >>>
>> >>> Hairong
>> >>>
>> >>> On 11/17/10 11:57 AM, "Todd Lipcon" <todd@cloudera.com> wrote:
>> >>>
>> >>> Hey all,
>> >>>
>> >>> As many of you know, I've been working on HDFS-1073 for a few months,
>> and
>> >>> we're hoping to get it in for 0.22 (OOM has given the OK to merge this
>> into
>> >>> branch and Sanjay has helpfully gotten some commitment from Yahoo QA
>> to help
>> >>> test it).
>> >>>
>> >>> It's been difficult recently to work on the patch as a lot of changes
>> have
>> >>> gone in around FSImage and FSEditLog (eg checksumming, parallel load,
>> etc)
>> >>> so the patch falls out of date quickly and is hard to resolve the
>> changes
>> >>> since my patch moves a lot of code. Ivan Kelly over at Yahoo Barcelona
>> has
>> >>> also been working on a refactor and I think running into a similar
>> issue.
>> >>>
>> >>> So, I just wanted to ping everyone who has been working on this area
>> of
>> >>> the code to find out if there are any more remaining JIRAs that you're
>> >>> hoping to commit in the next several weeks that will touch FSImage and
>> >>> FSEditLog. If so, we should try to agree on an order in which to
>> commit them
>> >>> so we don't end up stomping on each other, and we can help review each
>> >>> other's work to move quicker.
>> >>>
>> >>> If you can just reply with a list of any remaining uncommitted JIRAs
>> that
>> >>> touch FSImage/FSEditLog that would be great, then we can work from
>> there on
>> >>> how to proceed and minimize the amount of patch rebasing we all have
>> to do.
>> >>>
>> >>> Thanks!
>> >>> -Todd
>> >>
>> >>
>> >>
>> >> --
>> >> Todd Lipcon
>> >> Software Engineer, Cloudera
>> >>
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message