hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Shvachko <shv.had...@gmail.com>
Subject Re: [Vote] Merge branch-trunk-win to trunk
Date Tue, 05 Mar 2013 01:00:55 GMT
+1 on the merge.

I am glad we agreed.
Having Jira to track the CI effort is a good idea.

Thanks,
--Konstantin

On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mfoley@hortonworks.com> wrote:
> Thanks.  I agree Windows -1's in test-patch should not block commits.
>
> --Matt
>
>
>
> On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <shv.hadoop@gmail.com>
> wrote:
>>
>> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com>
>> wrote:
>> > Konstantine, you have voted -1, and stated some requirements before
>> > you'll
>> > withdraw that -1.  As I plan to do work to fulfill those requirements, I
>> > want to make sure that what I'm proposing will, in fact, satisfy you.
>> > That's why I'm asking, if we implement full "test-patch" integration for
>> > Windows, does it seem to you that that would provide adequate support?
>>
>> Yes.
>>
>> > I have learned not to presume that my interpretation is correct.  My
>> > interpretation of item #1 is that test-patch provides pre-commit build,
>> > so
>> > it would satisfy item #1.  But rather than assuming that I am
>> > interpreting
>> > it correctly, I simply want your agreement that it would, or if not,
>> > clarification why it won't.
>>
>> I agree it will satisfy my item #1.
>> I did not agree in my previous email, but I changed my mind based on
>> the latest discussion. I have to explain why now.
>> I was proposing nightly build because I did not want pre-commit build
>> for Windows block commits to Linux. But if people are fine just ignoring
>> -1s for the Windows part of the build it should be good.
>>
>> > Regarding item #2, it is also my interpretation that test-patch provides
>> > an
>> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit test,
>> > with
>> > logs available to the developer, so it would satisfy item #2.  But
>> > rather
>> > than assuming that I am interpreting it correctly, I simply want your
>> > agreement that it would, or if not, clarification why it won't.
>>
>> It will satisfy my item #2 in the following way:
>> I can duplicate your pre-commit build for Windows and add an input
>> parameter, which would let people run the build on their patches
>> chosen from local machine rather than attaching them to Jiras.
>>
>> Thanks,
>> --Konstantin
>>
>> > In agile terms, you are the Owner of these requirements.  Please give me
>> > owner feedback as to whether my proposed work sounds like it will
>> > satisfy
>> > the requirements.
>> >
>> > Thank you,
>> > --Matt
>> >
>> >
>> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > <shv.hadoop@gmail.com>
>> > wrote:
>> >>
>> >> Didn't I explain in details what I am asking for?
>> >>
>> >> Thanks,
>> >> --Konst
>> >>
>> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <mfoley@hortonworks.com>
>> >> wrote:
>> >> > Hi Konstantin,
>> >> > I'd like to point out two things:
>> >> > First, I already committed in this thread (email of Thu, Feb 28, 2013
>> >> > at
>> >> > 6:01 PM) to providing CI for Windows builds.  So please stop acting
>> >> > like
>> >> > I'm
>> >> > resisting this idea or something.
>> >> > Second, you didn't answer my question, you just kvetched about the
>> >> > phrasing.
>> >> > So I ask again:
>> >> >
>> >> > Will providing full "test-patch" integration (pre-commit build and
>> >> > unit
>> >> > test
>> >> > triggered by Jira "Patch Available" state) satisfy your request for
>> >> > functionality #1 and #2?  Yes or no, please.
>> >> >
>> >> > Thanks,
>> >> > --Matt
>> >> >
>> >> >
>> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> >> > <shv.hadoop@gmail.com>
>> >> > wrote:
>> >> >>
>> >> >> Hi Matt,
>> >> >>
>> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <mfoley@hortonworks.com>
>> >> >> wrote:
>> >> >> > Konstantin,
>> >> >> > I would like to explore what it would take to remove this
>> >> >> > perceived
>> >> >> > impediment --
>> >> >>
>> >> >> Glad you decided to explore. Thank you.
>> >> >>
>> >> >> > although I reserve the right to argue that this is not
>> >> >> > pre-requisite to merging the cross-platform support patch.
>> >> >>
>> >> >> It's your right indeed. So as mine to question what the platform
>> >> >> support means for you, which I believe remained unclear.
>> >> >> I do not impede the change as you should have noticed. My
>> >> >> requirement
>> >> >> comes from my perception of the support, which means to me exactly
>> >> >> two
>> >> >> things:
>> >> >> 1. The ability to recognise the code is broken for the platform
>> >> >> 2. The ability to test new patches on the platform
>> >> >> The latter is problematic, as many noticed in this thread, for
those
>> >> >> whose customary environment does not include Windows.
>> >> >>
>> >> >> > If we implemented full "test-patch" support for Windows on
trunk,
>> >> >> > would
>> >> >> > that
>> >> >> > fulfill both your items #1 and #2?  Please note that:
>> >> >> > a) Pushing the "Patch Available" button in Jira shall cause
a
>> >> >> > pre-commit
>> >> >> > build to start within, I believe, 20 minutes.
>> >> >> > b) That build keeps logs for both java build and unit tests
for
>> >> >> > several
>> >> >> > days, that are accessible to all viewers.
>> >> >>
>> >> >> In item #1 I mostly asking for the nightly build, which is simpler
>> >> >> than "test-patch". The latter would be ideal from the platform
>> >> >> support
>> >> >> viewpoint, but it is for the community to decide if we want to
add
>> >> >> extra +3 hours to the build.
>> >> >> Nightly build in my understanding is triggered by the timer rather
>> >> >> than by Jira's "submit patch" button.  On Jenkins build
>> >> >> configuration
>> >> >> you can specify it under "Build periodically".
>> >> >>
>> >> >> > So, does this provide sufficient on-demand support that we
don't
>> >> >> > have
>> >> >> > to
>> >> >> > implement a whole new on-demand VM support structure of some
sort
>> >> >> > for
>> >> >> > #2
>> >> >> > (which would be an extraordinary and impractical demand)?
>> >> >>
>> >> >> I did not mention VMs. Item #2 means a build, which runs
>> >> >> "test-patch"
>> >> >> target with the file specified by a user (instead of a jira
>> >> >> attachment).
>> >> >> When user clicks "Build Now" link a box is displayed where the
user
>> >> >> can enter the file path containing the patch. This can be specified
>> >> >> in
>> >> >> the Build Configuration under "This build is parameterized" by
>> >> >> choosing AddParameter / FileParameter. The build can run on the
same
>> >> >> Windows machine as the nightly build.
>> >> >> Such build will let people test their patches for Windows on Jenkins
>> >> >> if they don't posses a license for the right version of Windows.
>> >> >> I hope this will not turn into extraordinary or impractical effort.
>> >> >>
>> >> >> Thanks,
>> >> >> --Konst
>> >> >>
>> >> >> > Thanks,
>> >> >> > --Matt
>> >> >> >
>> >> >> >
>> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
>> >> >> > <shv.hadoop@gmail.com>
>> >> >> > wrote:
>> >> >> >>
>> >> >> >> -1
>> >> >> >> We should have a CI infrastructure in place before we
can commit
>> >> >> >> to
>> >> >> >> supporting Windows platform.
>> >> >> >>
>> >> >> >> Eric is right Win/Cygwin was supported since day one.
>> >> >> >> I had a Windows box under my desk running nightly builds
back in
>> >> >> >> 2006-07.
>> >> >> >> People were irritated but I was filing windows bugs until
0.22
>> >> >> >> release.
>> >> >> >> Times changing and I am glad to see wider support for
Win
>> >> >> >> platform.
>> >> >> >>
>> >> >> >> But in order to make it work you guys need to put the
CI process
>> >> >> >> in
>> >> >> >> place
>> >> >> >>
>> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
>> >> >> >> - Nightly would mean that changes can be committed to
trunk based
>> >> >> >> on
>> >> >> >> linux PreCommit build. And people will file bugs if the
change
>> >> >> >> broke
>> >> >> >> Windows nightly build.
>> >> >> >> - PreCommit-win build will mean automatic reporting failed
tests
>> >> >> >> to
>> >> >> >> respective jira blocking commits the same way as it is
now with
>> >> >> >> linux
>> >> >> >> PreCommit builds.
>> >> >> >> We should discuss which way is more efficient for developers.
>> >> >> >>
>> >> >> >> 2. On-demand-windows Jenkins build.
>> >> >> >> I see it as a build to which I can attach my patch and
the build
>> >> >> >> will
>> >> >> >> run my changes on a dedicated windows box.
>> >> >> >> That way people can test their changes without having
personal
>> >> >> >> windows
>> >> >> >> nodes.
>> >> >> >>
>> >> >> >> I think this is the minimal set of requirement for us
to be able
>> >> >> >> to
>> >> >> >> commit to the new platform.
>> >> >> >> Right now I see only one windows related build
>> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> >> >> >> Which was failing since Sept 8, 2012 and did not run in
the last
>> >> >> >> month.
>> >> >> >>
>> >> >> >> Thanks,
>> >> >> >> --Konst
>> >> >> >>
>> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
>> >> >> >> <eric14@hortonworks.com> wrote:
>> >> >> >> > +1 (non-binding)
>> >> >> >> >
>> >> >> >> > A few of observations:
>> >> >> >> >
>> >> >> >> > - Windows has actually been a supported platform
for Hadoop
>> >> >> >> > since
>> >> >> >> > 0.1
>> >> >> >> > .
>> >> >> >> > Doug championed supporting windows then and we've
continued to
>> >> >> >> > do
>> >> >> >> > it
>> >> >> >> > with
>> >> >> >> > varying vigor over time.  To my knowledge we've never
made a
>> >> >> >> > decision
>> >> >> >> > to
>> >> >> >> > drop windows support.  The change here is improving
our support
>> >> >> >> > and
>> >> >> >> > dropping
>> >> >> >> > the requirement of cigwin.  We had Nutch windows
users on the
>> >> >> >> > list
>> >> >> >> > in
>> >> >> >> > 2006
>> >> >> >> > and we've been supporting windows FS requirements
since
>> >> >> >> > inception.
>> >> >> >> >
>> >> >> >> > - A little pragmatism will go a long way.  As a community
we've
>> >> >> >> > got
>> >> >> >> > to
>> >> >> >> > stay committed to keeping hadoop simple (so it does
work on
>> >> >> >> > many
>> >> >> >> > platforms)
>> >> >> >> > and extending it to take advantage of key emerging
OS/hardware
>> >> >> >> > features,
>> >> >> >> > such as containers, new FSs, virtualization, flash
...  We
>> >> >> >> > should
>> >> >> >> > all
>> >> >> >> > plan
>> >> >> >> > to let new features & optimizations emerge that
don't work
>> >> >> >> > everywhere, if
>> >> >> >> > they are compelling and central to hadoop's mission
of being
>> >> >> >> > THE
>> >> >> >> > best
>> >> >> >> > fabric
>> >> >> >> > for storing and processing big data.
>> >> >> >> >
>> >> >> >> > - A UI project like KDE has to deal with the MANY
differences
>> >> >> >> > between
>> >> >> >> > windows and linux UI APIs.  Hadoop faces no such
complex
>> >> >> >> > challenge
>> >> >> >> > and hence
>> >> >> >> > can be maintained from a single codeline IMO.  It
is mostly
>> >> >> >> > abstracted from
>> >> >> >> > the OS APIs via Java and our design choices.  Where
it is not
>> >> >> >> > we
>> >> >> >> > can
>> >> >> >> > continue to add plugable abstractions.
>> >> >> >> >
>> >> >> >
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>

Mime
View raw message