Mailing-List: contact mapreduce-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of mfoley@hortonworks.com
 designates 209.85.217.176 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKtuutHdE1xOLOGGC0mz_aoKvhpf1oPwn8Lmc9M-9NhbOkmhBg@mail.gmail.com>
References: 
 <CABCYYb_nmDWRf_3y6GQ+U+E2zwghfZ6AraerJ-eEz4m5i5ySLA@mail.gmail.com>
 <CD54EB89.6FF2%evans@yahoo-inc.com>
 <CABCYYb8wbiNvNsHdSRK+wHMeKMmhSN+UHctTD=bTtsjay=ab1w@mail.gmail.com>
 <ac1f890c4f3c42ff823ebf39f8b72dd6@BLUPR03MB614.namprd03.prod.outlook.com>
 <CA+z3+9F+JrS-_+x_BfpBppx5SsRTEwY3ZPJm0449h1yrnEsfLg@mail.gmail.com>
 <CA+z3+9EfF5VyXqDPyP71tKY4N25GpKj2HY9Hs1O0KTQ+fkfV3w@mail.gmail.com>
 <CA+z3+9H=i5cWfBBMRN5uy4WaC_rF56sakHcpcYc=gzqwYxf5Vw@mail.gmail.com>
 <FA58897E-8389-492F-800E-59DB30EF95D9@hortonworks.com>
 <CAKtuutGg4kv8aZDz0awb-0N1Z0c7sbOcgFY2MP1MaKYu00TStQ@mail.gmail.com>
 <CA+z3+9EW_Jj2LV0_anEhAwZraLXZHvXnQj9bKQpbJZcqbbE4wg@mail.gmail.com>
 <CAKtuutGc1VKovv_JpRCjC7xx989G-N=0JdSSTd2XxD==Y1Kshg@mail.gmail.com>
 <CA+z3+9EFgvEXH+0xNBTL_TK40+FZAtG+SaZ0GMj+=JsBRUf7=A@mail.gmail.com>
 <CAKtuutHO_P0cesJNednhPwNK=_zeYkB2xa3MyoU3X3=Fztdeng@mail.gmail.com>
 <CA+z3+9H1uw_C-=dWmcEk_p6Yi6S8oqorKHa+F1Wg0W40ReF_pg@mail.gmail.com>
 <CAKtuutG=86QL11dPySFwFd=XU_hE6dxJZ0a0SuZS=-t9ZwPKPw@mail.gmail.com>
 <CA+z3+9FXB2rxs+q2JTERVBiuUPHdvdGcSqJVYSqVif+Qq1GvxA@mail.gmail.com>
 <CAKtuutHdE1xOLOGGC0mz_aoKvhpf1oPwn8Lmc9M-9NhbOkmhBg@mail.gmail.com>
From: Matt Foley <mfoley@hortonworks.com>
Date: Mon, 4 Mar 2013 17:41:34 -0800
Message-ID: 
 <CA+z3+9GuGn+wb7yoCNL7P8urNNz0C6piadfHw2RuN=fB8X5ORQ@mail.gmail.com>
Subject: Re: [Vote] Merge branch-trunk-win to trunk
To: Konstantin Shvachko <shv.hadoop@gmail.com>
Cc: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>,
	"hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>,
 mapreduce-dev@hadoop.apache.org,
	yarn-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=f46d0408da07ba9e6404d72394ba

--f46d0408da07ba9e6404d72394ba
Content-Type: text/plain; charset=ISO-8859-1

Thanks, gentlemen.  I've opened and taken responsibility for
https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has agreed
to help with the parts that require Jenkins admin access.

Thanks,
--Matt


On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <shv.hadoop@gmail.com>wrote:

> +1 on the merge.
>
> I am glad we agreed.
> Having Jira to track the CI effort is a good idea.
>
> Thanks,
> --Konstantin
>
> On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mfoley@hortonworks.com> wrote:
> > Thanks.  I agree Windows -1's in test-patch should not block commits.
> >
> > --Matt
> >
> >
> >
> > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>
> > wrote:
> >>
> >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com>
> >> wrote:
> >> > Konstantine, you have voted -1, and stated some requirements before
> >> > you'll
> >> > withdraw that -1.  As I plan to do work to fulfill those
> requirements, I
> >> > want to make sure that what I'm proposing will, in fact, satisfy you.
> >> > That's why I'm asking, if we implement full "test-patch" integration
> for
> >> > Windows, does it seem to you that that would provide adequate support?
> >>
> >> Yes.
> >>
> >> > I have learned not to presume that my interpretation is correct.  My
> >> > interpretation of item #1 is that test-patch provides pre-commit
> build,
> >> > so
> >> > it would satisfy item #1.  But rather than assuming that I am
> >> > interpreting
> >> > it correctly, I simply want your agreement that it would, or if not,
> >> > clarification why it won't.
> >>
> >> I agree it will satisfy my item #1.
> >> I did not agree in my previous email, but I changed my mind based on
> >> the latest discussion. I have to explain why now.
> >> I was proposing nightly build because I did not want pre-commit build
> >> for Windows block commits to Linux. But if people are fine just ignoring
> >> -1s for the Windows part of the build it should be good.
> >>
> >> > Regarding item #2, it is also my interpretation that test-patch
> provides
> >> > an
> >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
> >> > with
> >> > logs available to the developer, so it would satisfy item #2.  But
> >> > rather
> >> > than assuming that I am interpreting it correctly, I simply want your
> >> > agreement that it would, or if not, clarification why it won't.
> >>
> >> It will satisfy my item #2 in the following way:
> >> I can duplicate your pre-commit build for Windows and add an input
> >> parameter, which would let people run the build on their patches
> >> chosen from local machine rather than attaching them to Jiras.
> >>
> >> Thanks,
> >> --Konstantin
> >>
> >> > In agile terms, you are the Owner of these requirements.  Please give
> me
> >> > owner feedback as to whether my proposed work sounds like it will
> >> > satisfy
> >> > the requirements.
> >> >
> >> > Thank you,
> >> > --Matt
> >> >
> >> >
> >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> >> > <shv.hadoop@gmail.com>
> >> > wrote:
> >> >>
> >> >> Didn't I explain in details what I am asking for?
> >> >>
> >> >> Thanks,
> >> >> --Konst
> >> >>
> >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mfoley@hortonworks.com>
> >> >> wrote:
> >> >> > Hi Konstantin,
> >> >> > I'd like to point out two things:
> >> >> > First, I already committed in this thread (email of Thu, Feb 28,
> 2013
> >> >> > at
> >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
> >> >> > like
> >> >> > I'm
> >> >> > resisting this idea or something.
> >> >> > Second, you didn't answer my question, you just kvetched about the
> >> >> > phrasing.
> >> >> > So I ask again:
> >> >> >
> >> >> > Will providing full "test-patch" integration (pre-commit build and
> >> >> > unit
> >> >> > test
> >> >> > triggered by Jira "Patch Available" state) satisfy your request for
> >> >> > functionality #1 and #2?  Yes or no, please.
> >> >> >
> >> >> > Thanks,
> >> >> > --Matt
> >> >> >
> >> >> >
> >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> >> >> > <shv.hadoop@gmail.com>
> >> >> > wrote:
> >> >> >>
> >> >> >> Hi Matt,
> >> >> >>
> >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> mfoley@hortonworks.com>
> >> >> >> wrote:
> >> >> >> > Konstantin,
> >> >> >> > I would like to explore what it would take to remove this
> >> >> >> > perceived
> >> >> >> > impediment --
> >> >> >>
> >> >> >> Glad you decided to explore. Thank you.
> >> >> >>
> >> >> >> > although I reserve the right to argue that this is not
> >> >> >> > pre-requisite to merging the cross-platform support patch.
> >> >> >>
> >> >> >> It's your right indeed. So as mine to question what the platform
> >> >> >> support means for you, which I believe remained unclear.
> >> >> >> I do not impede the change as you should have noticed. My
> >> >> >> requirement
> >> >> >> comes from my perception of the support, which means to me exactly
> >> >> >> two
> >> >> >> things:
> >> >> >> 1. The ability to recognise the code is broken for the platform
> >> >> >> 2. The ability to test new patches on the platform
> >> >> >> The latter is problematic, as many noticed in this thread, for
> those
> >> >> >> whose customary environment does not include Windows.
> >> >> >>
> >> >> >> > If we implemented full "test-patch" support for Windows on
> trunk,
> >> >> >> > would
> >> >> >> > that
> >> >> >> > fulfill both your items #1 and #2?  Please note that:
> >> >> >> > a) Pushing the "Patch Available" button in Jira shall cause a
> >> >> >> > pre-commit
> >> >> >> > build to start within, I believe, 20 minutes.
> >> >> >> > b) That build keeps logs for both java build and unit tests for
> >> >> >> > several
> >> >> >> > days, that are accessible to all viewers.
> >> >> >>
> >> >> >> In item #1 I mostly asking for the nightly build, which is simpler
> >> >> >> than "test-patch". The latter would be ideal from the platform
> >> >> >> support
> >> >> >> viewpoint, but it is for the community to decide if we want to add
> >> >> >> extra +3 hours to the build.
> >> >> >> Nightly build in my understanding is triggered by the timer rather
> >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> >> >> >> configuration
> >> >> >> you can specify it under "Build periodically".
> >> >> >>
> >> >> >> > So, does this provide sufficient on-demand support that we don't
> >> >> >> > have
> >> >> >> > to
> >> >> >> > implement a whole new on-demand VM support structure of some
> sort
> >> >> >> > for
> >> >> >> > #2
> >> >> >> > (which would be an extraordinary and impractical demand)?
> >> >> >>
> >> >> >> I did not mention VMs. Item #2 means a build, which runs
> >> >> >> "test-patch"
> >> >> >> target with the file specified by a user (instead of a jira
> >> >> >> attachment).
> >> >> >> When user clicks "Build Now" link a box is displayed where the
> user
> >> >> >> can enter the file path containing the patch. This can be
> specified
> >> >> >> in
> >> >> >> the Build Configuration under "This build is parameterized" by
> >> >> >> choosing AddParameter / FileParameter. The build can run on the
> same
> >> >> >> Windows machine as the nightly build.
> >> >> >> Such build will let people test their patches for Windows on
> Jenkins
> >> >> >> if they don't posses a license for the right version of Windows.
> >> >> >> I hope this will not turn into extraordinary or impractical
> effort.
> >> >> >>
> >> >> >> Thanks,
> >> >> >> --Konst
> >> >> >>
> >> >> >> > Thanks,
> >> >> >> > --Matt
> >> >> >> >
> >> >> >> >
> >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> >> >> >> > <shv.hadoop@gmail.com>
> >> >> >> > wrote:
> >> >> >> >>
> >> >> >> >> -1
> >> >> >> >> We should have a CI infrastructure in place before we can
> commit
> >> >> >> >> to
> >> >> >> >> supporting Windows platform.
> >> >> >> >>
> >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> >> >> >> >> I had a Windows box under my desk running nightly builds back
> in
> >> >> >> >> 2006-07.
> >> >> >> >> People were irritated but I was filing windows bugs until 0.22
> >> >> >> >> release.
> >> >> >> >> Times changing and I am glad to see wider support for Win
> >> >> >> >> platform.
> >> >> >> >>
> >> >> >> >> But in order to make it work you guys need to put the CI
> process
> >> >> >> >> in
> >> >> >> >> place
> >> >> >> >>
> >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> >> >> >> >> - Nightly would mean that changes can be committed to trunk
> based
> >> >> >> >> on
> >> >> >> >> linux PreCommit build. And people will file bugs if the change
> >> >> >> >> broke
> >> >> >> >> Windows nightly build.
> >> >> >> >> - PreCommit-win build will mean automatic reporting failed
> tests
> >> >> >> >> to
> >> >> >> >> respective jira blocking commits the same way as it is now with
> >> >> >> >> linux
> >> >> >> >> PreCommit builds.
> >> >> >> >> We should discuss which way is more efficient for developers.
> >> >> >> >>
> >> >> >> >> 2. On-demand-windows Jenkins build.
> >> >> >> >> I see it as a build to which I can attach my patch and the
> build
> >> >> >> >> will
> >> >> >> >> run my changes on a dedicated windows box.
> >> >> >> >> That way people can test their changes without having personal
> >> >> >> >> windows
> >> >> >> >> nodes.
> >> >> >> >>
> >> >> >> >> I think this is the minimal set of requirement for us to be
> able
> >> >> >> >> to
> >> >> >> >> commit to the new platform.
> >> >> >> >> Right now I see only one windows related build
> >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> >> >> >> Which was failing since Sept 8, 2012 and did not run in the
> last
> >> >> >> >> month.
> >> >> >> >>
> >> >> >> >> Thanks,
> >> >> >> >> --Konst
> >> >> >> >>
> >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> >> >> >> >> <eric14@hortonworks.com> wrote:
> >> >> >> >> > +1 (non-binding)
> >> >> >> >> >
> >> >> >> >> > A few of observations:
> >> >> >> >> >
> >> >> >> >> > - Windows has actually been a supported platform for Hadoop
> >> >> >> >> > since
> >> >> >> >> > 0.1
> >> >> >> >> > .
> >> >> >> >> > Doug championed supporting windows then and we've continued
> to
> >> >> >> >> > do
> >> >> >> >> > it
> >> >> >> >> > with
> >> >> >> >> > varying vigor over time.  To my knowledge we've never made a
> >> >> >> >> > decision
> >> >> >> >> > to
> >> >> >> >> > drop windows support.  The change here is improving our
> support
> >> >> >> >> > and
> >> >> >> >> > dropping
> >> >> >> >> > the requirement of cigwin.  We had Nutch windows users on the
> >> >> >> >> > list
> >> >> >> >> > in
> >> >> >> >> > 2006
> >> >> >> >> > and we've been supporting windows FS requirements since
> >> >> >> >> > inception.
> >> >> >> >> >
> >> >> >> >> > - A little pragmatism will go a long way.  As a community
> we've
> >> >> >> >> > got
> >> >> >> >> > to
> >> >> >> >> > stay committed to keeping hadoop simple (so it does work on
> >> >> >> >> > many
> >> >> >> >> > platforms)
> >> >> >> >> > and extending it to take advantage of key emerging
> OS/hardware
> >> >> >> >> > features,
> >> >> >> >> > such as containers, new FSs, virtualization, flash ...  We
> >> >> >> >> > should
> >> >> >> >> > all
> >> >> >> >> > plan
> >> >> >> >> > to let new features & optimizations emerge that don't work
> >> >> >> >> > everywhere, if
> >> >> >> >> > they are compelling and central to hadoop's mission of being
> >> >> >> >> > THE
> >> >> >> >> > best
> >> >> >> >> > fabric
> >> >> >> >> > for storing and processing big data.
> >> >> >> >> >
> >> >> >> >> > - A UI project like KDE has to deal with the MANY differences
> >> >> >> >> > between
> >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such complex
> >> >> >> >> > challenge
> >> >> >> >> > and hence
> >> >> >> >> > can be maintained from a single codeline IMO.  It is mostly
> >> >> >> >> > abstracted from
> >> >> >> >> > the OS APIs via Java and our design choices.  Where it is not
> >> >> >> >> > we
> >> >> >> >> > can
> >> >> >> >> > continue to add plugable abstractions.
> >> >> >> >> >
> >> >> >> >
> >> >> >> >
> >> >> >
> >> >> >
> >> >
> >> >
> >
> >
>

--f46d0408da07ba9e6404d72394ba--