hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eli Collins <...@cloudera.com>
Subject Re: Changes to FSImage/FSEditLog
Date Sun, 28 Nov 2010 22:01:24 GMT
Hey Todd, Ivan,

I've committed the patch for 1473.

Thanks,
Eli

On Wed, Nov 24, 2010 at 2:25 PM, Todd Lipcon <todd@cloudera.com> wrote:
> Hi all,
>
> There's a patch up for 1473 now. If we can get a +1 we can commit this and
> move on to the next steps in some of these projects.
>
> Thanks
> -Todd
>
> On Thu, Nov 18, 2010 at 12:42 AM, Ivan Kelly <ivank@yahoo-inc.com> wrote:
>
>> This patch looks really good. It shouldn't affect our refactoring too much
>> and will actually make things a lot easier for the next part (breaking the
>> circular dependency between namesystem and fsimage).
>>
>> -Ivan
>>
>>
>> On 18 Nov 2010, at 02:27, Todd Lipcon wrote:
>>
>> I put up a sketch of a patch for 1473. It compiles and passes at least a
>> couple of the unit tests, but more work to be done on it. Would appreciate
>> comments from this crowd if the new classes seem like good divisions.
>>
>> -Todd
>>
>> On Wed, Nov 17, 2010 at 2:36 PM, Todd Lipcon <todd@cloudera.com> wrote:
>>
>>> On Wed, Nov 17, 2010 at 2:34 PM, Hairong Kuang <kuang.hairong@fb.com>wrote:
>>>
>>>> You meant HDFS-1473 right? When do you expect that it could be done?
>>>>
>>>>
>>> I was working on HDFS-1473 this morning and could probably get a
>>> preliminary patch up today.. but when I realized that we have 4 parallel
>>> refactorings going on I paused my work. If we agree that HDFS-1473 will go
>>> in before the other ones I can finish it up, but if we're doing the same
>>> thing we should coordinate instead.
>>>
>>>
>>>> Hairong
>>>>
>>>> On 11/17/10 2:29 PM, "Eli Collins" <eli@cloudera.com> wrote:
>>>>
>>>> > Agree we should do the refactoring before the features, we should also
>>>> > add the tests with the refactoring to show they don't break things.
>>>> >
>>>> > On Wed, Nov 17, 2010 at 2:27 PM, Todd Lipcon <todd@cloudera.com>
>>>> wrote:
>>>> >> OK, looks like there is still quite a bit more in motion here, thanks
>>>> for
>>>> >> the list, Hairong.
>>>> >> How would everyone feel about setting aside those improvements/fixes,
>>>> doing
>>>> >> the refactor first, and then going back to them? It should help
with
>>>> testing
>>>> >> and reduce conflicts between the various ongoing projects.
>>>> >> Maybe we can discuss quickly on this thread: what are some small
>>>> refactors
>>>> >> we can start with to help separate these things? Ivan's project
looks
>>>> really
>>>> >> good but it seems like it is a pretty large change. If we can break
it
>>>> up
>>>> >> into a couple smaller patches we can commit as we go.
>>>> >> -Todd
>>>> >> On Wed, Nov 17, 2010 at 2:08 PM, Hairong Kuang <kuang.hairong@fb.com>
>>>> wrote:
>>>> >>>
>>>> >>> Hi Todd,
>>>> >>>
>>>> >>> Here is the list of fsimage jiras that I am working on:
>>>> >>>
>>>> >>> HDFS-1481 // small change
>>>> >>> HDFS-1458 // small change
>>>> >>> HDFS-1496  // should be checked into 022, I still have no clue
how to
>>>> fix
>>>> >>> it
>>>> >>> HDFS-1070  // this one has a simple idea, but turns out that
it needs
>>>> >>> major code re-organization. I am still struggling with how to
make my
>>>> code
>>>> >>> change easier to review. But this will definitely collide with
yours.
>>>> >>>
>>>> >>> Hairong
>>>> >>>
>>>> >>> On 11/17/10 11:57 AM, "Todd Lipcon" <todd@cloudera.com>
wrote:
>>>> >>>
>>>> >>> Hey all,
>>>> >>>
>>>> >>> As many of you know, I've been working on HDFS-1073 for a few
months,
>>>> and
>>>> >>> we're hoping to get it in for 0.22 (OOM has given the OK to
merge
>>>> this into
>>>> >>> branch and Sanjay has helpfully gotten some commitment from
Yahoo QA
>>>> to help
>>>> >>> test it).
>>>> >>>
>>>> >>> It's been difficult recently to work on the patch as a lot of
changes
>>>> have
>>>> >>> gone in around FSImage and FSEditLog (eg checksumming, parallel
load,
>>>> etc)
>>>> >>> so the patch falls out of date quickly and is hard to resolve
the
>>>> changes
>>>> >>> since my patch moves a lot of code. Ivan Kelly over at Yahoo
>>>> Barcelona has
>>>> >>> also been working on a refactor and I think running into a similar
>>>> issue.
>>>> >>>
>>>> >>> So, I just wanted to ping everyone who has been working on this
area
>>>> of
>>>> >>> the code to find out if there are any more remaining JIRAs that
>>>> you're
>>>> >>> hoping to commit in the next several weeks that will touch FSImage
>>>> and
>>>> >>> FSEditLog. If so, we should try to agree on an order in which
to
>>>> commit them
>>>> >>> so we don't end up stomping on each other, and we can help review
>>>> each
>>>> >>> other's work to move quicker.
>>>> >>>
>>>> >>> If you can just reply with a list of any remaining uncommitted
JIRAs
>>>> that
>>>> >>> touch FSImage/FSEditLog that would be great, then we can work
from
>>>> there on
>>>> >>> how to proceed and minimize the amount of patch rebasing we
all have
>>>> to do.
>>>> >>>
>>>> >>> Thanks!
>>>> >>> -Todd
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Todd Lipcon
>>>> >> Software Engineer, Cloudera
>>>> >>
>>>>
>>>>
>>>
>>>
>>> --
>>> Todd Lipcon
>>> Software Engineer, Cloudera
>>>
>>
>>
>>
>> --
>> Todd Lipcon
>> Software Engineer, Cloudera
>>
>>
>>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Mime
View raw message