Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
MIME-Version: 1.0
In-Reply-To: <CAGB5D2amLJGbrnaB-AwtfN-k6pPoXuQLvyK5XXe_SDu0uEZZJQ@mail.gmail.com>
References: <CAD++eCnjEHU=-M33QdjnND0ZL73eKwxRua4=Bbp4G8inQZmaMg@mail.gmail.com>
 <CAD++eCkG_AV_7sFFLDYF2OYzXKQmYD7CWm3cO8mvfJz5rawbTw@mail.gmail.com> <CAGB5D2amLJGbrnaB-AwtfN-k6pPoXuQLvyK5XXe_SDu0uEZZJQ@mail.gmail.com>
From: Wangda Tan <wheeleast@gmail.com>
Date: Fri, 18 Aug 2017 15:05:18 -0700
Message-ID: <CAD++eC=x48cqeWd4pw1aU=-FCXj99EmnqheeJRRDyUdytzmqOQ@mail.gmail.com>
Subject: Re: [DISCUSS] Merge YARN resource profile (YARN-3926) branch into trunk
To: Andrew Wang <andrew.wang@cloudera.com>
Cc: "yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>,
	"hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>,
	"common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>,
	"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary="f403045c0c8ce4955805570e5407"
archived-at: Fri, 18 Aug 2017 22:06:13 -0000

--f403045c0c8ce4955805570e5407
Content-Type: text/plain; charset="UTF-8"

Hi Andrew,

This feature can be disabled, It is off by default. The major change of
existing code path it Resource YARN PB record related implementations, we
did lots of tests around this regarding to performances and safety of these
changes, so far so good (please refer to my email regarding to performance
and compatibility). Beyond resource PB implementation changes, it mostly
add new code path instead of modifying existing logics. We will continue to
do more verifications next week to minimize risks.

As I mentioned, new added fields are marked as Unstable right now (and we
will convert some @evolving to  @unstable). They're all stated in javadocs.

I completely understand the recent emerging merge request makes everybody
nervous, we will try to do more tests and move fast to meet the time line.
:)

Please let me know if you have any other concerns.

Thanks,
Wangda


On Fri, Aug 18, 2017 at 2:34 PM, Andrew Wang <andrew.wang@cloudera.com>
wrote:

> Hi Wangda,
>
> Can this feature be disabled? Is it on or off by default? We're 1 month
> from the target release for beta1, so I don't want to introduce risk to
> existing code paths. TSv2 and S3Guard and YARN Federation are all okay in
> that regard.
>
> I'm also not clear on what work is remaining, there are a lot of
> unresolved subtasks still under YARN-3926.
>
> In terms of compatibility, the design doc talks about some PB changes. Are
> these stable? Is there documentation somewhere that explains what APIs are
> stable or unstable for users?
>
> I'm going to start a separate discussion about beta1 scope. I think this
> is the fourth merge proposal this week, and this amount of code movement
> makes me very nervous considering that beta1 is only a month out.
>
> Best,
> Andrew
>
> On Thu, Aug 17, 2017 at 8:27 PM, Wangda Tan <wheeleast@gmail.com> wrote:
>
>> +hdfs/common/mr
>>
>> On Thu, Aug 17, 2017 at 1:28 PM, Wangda Tan <wheeleast@gmail.com> wrote:
>>
>> > Hi all,
>> >
>> > I want to hear your thoughts of merging YARN resource profile branch
>> into
>> > trunk in the next few weeks. The goal is to get it in for Hadoop 3.0
>> beta1.
>> >
>> > *Regarding to testing:*
>> > We did extensive tests for the feature in the last several months.
>> > Comparing to latest trunk.
>> > - For SLS benchmark: We didn't see observable performance gap from
>> > simulated test based on 8K nodes SLS traces (1 PB memory). We got 3k+
>> > containers allocated per second.
>> > - For microbenchmark: We use performance test cases added by YARN-6775,
>> it
>> > shows around 5% performance regression comparing to trunk.
>> >
>> > *Regarding to API stability: *
>> > Most new added @Public APIs are @Unstable (We're going to convert some
>> new
>> > added @Public/@Evolving to @Unstable in the cleanup JIRA as well), we
>> want
>> > to get this included by beta1 so we get some feedbacks before declaring
>> > stable API.
>> >
>> > There're few pending cleanups under YARN-3926 umbrella JIRA. Besides
>> these
>> > cleanups, this feature works from end-to-end, we will do another
>> iteration
>> > of end-to-end tests after cleanup patches got committed.
>> >
>> > We would love to get your thoughts before opening a voting thread.
>> >
>> > Special thanks to a team of folks who worked hard and contributed
>> towards
>> > this efforts including design discussion / patch / reviews, etc.: Varun
>> > Vasudev, Sunil Govind, Daniel Templeton, Vinod Vavilapalli, Yufei Gu,
>> > Karthik Kambatla, Jason Lowe, Arun Suresh.
>> >
>> > Thanks,
>> > Wangda Tan
>> >
>>
>
>

--f403045c0c8ce4955805570e5407--