hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Shvachko <shv.had...@gmail.com>
Subject Re: [Vote] Merge branch-trunk-win to trunk
Date Mon, 25 Mar 2013 21:25:38 GMT
Andrew, this used to be on all -dev lists. Let's keep it that way.

To the point.
Does this mean that people are silently porting windows changes to branch-2?
New features on a branch should be voted first, no?

Thanks,
--Konstantin


On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <apurtell@apache.org> wrote:
> Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> how this could not have been caught prior to check-in.
>
>
> On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <cos@apache.org> wrote:
>
>> It doesn't look like any progress has been done on the ticket below in the
>> last 3 weeks. And now branch-2 can't be compiled because of
>>
>>
>> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
>> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
>> outside package
>>
>> That's exactly why I was -1'ing this...
>>   Cos
>>
>> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
>> > Thanks, gentlemen.  I've opened and taken responsibility for
>> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
>> agreed
>> > to help with the parts that require Jenkins admin access.
>> >
>> > Thanks,
>> > --Matt
>> >
>> >
>> >
>> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
>> shv.hadoop@gmail.com>wrote:
>> >
>> > > +1 on the merge.
>> > >
>> > > I am glad we agreed.
>> > > Having Jira to track the CI effort is a good idea.
>> > >
>> > > Thanks,
>> > > --Konstantin
>> > >
>> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mfoley@hortonworks.com>
>> wrote:
>> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
>> > > >
>> > > > --Matt
>> > > >
>> > > >
>> > > >
>> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
>> > > shv.hadoop@gmail.com>
>> > > > wrote:
>> > > >>
>> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
>> >
>> > > >> wrote:
>> > > >> > Konstantine, you have voted -1, and stated some requirements
>> before
>> > > >> > you'll
>> > > >> > withdraw that -1.  As I plan to do work to fulfill those
>> > > requirements, I
>> > > >> > want to make sure that what I'm proposing will, in fact,
satisfy
>> you.
>> > > >> > That's why I'm asking, if we implement full "test-patch"
>> integration
>> > > for
>> > > >> > Windows, does it seem to you that that would provide adequate
>> support?
>> > > >>
>> > > >> Yes.
>> > > >>
>> > > >> > I have learned not to presume that my interpretation is correct.
>>  My
>> > > >> > interpretation of item #1 is that test-patch provides pre-commit
>> > > build,
>> > > >> > so
>> > > >> > it would satisfy item #1.  But rather than assuming that
I am
>> > > >> > interpreting
>> > > >> > it correctly, I simply want your agreement that it would,
or if
>> not,
>> > > >> > clarification why it won't.
>> > > >>
>> > > >> I agree it will satisfy my item #1.
>> > > >> I did not agree in my previous email, but I changed my mind based
on
>> > > >> the latest discussion. I have to explain why now.
>> > > >> I was proposing nightly build because I did not want pre-commit
>> build
>> > > >> for Windows block commits to Linux. But if people are fine just
>> ignoring
>> > > >> -1s for the Windows part of the build it should be good.
>> > > >>
>> > > >> > Regarding item #2, it is also my interpretation that test-patch
>> > > provides
>> > > >> > an
>> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and
unit
>> test,
>> > > >> > with
>> > > >> > logs available to the developer, so it would satisfy item
#2.  But
>> > > >> > rather
>> > > >> > than assuming that I am interpreting it correctly, I simply
want
>> your
>> > > >> > agreement that it would, or if not, clarification why it
won't.
>> > > >>
>> > > >> It will satisfy my item #2 in the following way:
>> > > >> I can duplicate your pre-commit build for Windows and add an input
>> > > >> parameter, which would let people run the build on their patches
>> > > >> chosen from local machine rather than attaching them to Jiras.
>> > > >>
>> > > >> Thanks,
>> > > >> --Konstantin
>> > > >>
>> > > >> > In agile terms, you are the Owner of these requirements.
 Please
>> give
>> > > me
>> > > >> > owner feedback as to whether my proposed work sounds like
it will
>> > > >> > satisfy
>> > > >> > the requirements.
>> > > >> >
>> > > >> > Thank you,
>> > > >> > --Matt
>> > > >> >
>> > > >> >
>> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
>> > > >> > <shv.hadoop@gmail.com>
>> > > >> > wrote:
>> > > >> >>
>> > > >> >> Didn't I explain in details what I am asking for?
>> > > >> >>
>> > > >> >> Thanks,
>> > > >> >> --Konst
>> > > >> >>
>> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
>> mfoley@hortonworks.com>
>> > > >> >> wrote:
>> > > >> >> > Hi Konstantin,
>> > > >> >> > I'd like to point out two things:
>> > > >> >> > First, I already committed in this thread (email
of Thu, Feb
>> 28,
>> > > 2013
>> > > >> >> > at
>> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So
please stop
>> acting
>> > > >> >> > like
>> > > >> >> > I'm
>> > > >> >> > resisting this idea or something.
>> > > >> >> > Second, you didn't answer my question, you just
kvetched about
>> the
>> > > >> >> > phrasing.
>> > > >> >> > So I ask again:
>> > > >> >> >
>> > > >> >> > Will providing full "test-patch" integration (pre-commit
build
>> and
>> > > >> >> > unit
>> > > >> >> > test
>> > > >> >> > triggered by Jira "Patch Available" state) satisfy
your
>> request for
>> > > >> >> > functionality #1 and #2?  Yes or no, please.
>> > > >> >> >
>> > > >> >> > Thanks,
>> > > >> >> > --Matt
>> > > >> >> >
>> > > >> >> >
>> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
>> > > >> >> > <shv.hadoop@gmail.com>
>> > > >> >> > wrote:
>> > > >> >> >>
>> > > >> >> >> Hi Matt,
>> > > >> >> >>
>> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley
<
>> > > mfoley@hortonworks.com>
>> > > >> >> >> wrote:
>> > > >> >> >> > Konstantin,
>> > > >> >> >> > I would like to explore what it would take
to remove this
>> > > >> >> >> > perceived
>> > > >> >> >> > impediment --
>> > > >> >> >>
>> > > >> >> >> Glad you decided to explore. Thank you.
>> > > >> >> >>
>> > > >> >> >> > although I reserve the right to argue that
this is not
>> > > >> >> >> > pre-requisite to merging the cross-platform
support patch.
>> > > >> >> >>
>> > > >> >> >> It's your right indeed. So as mine to question
what the
>> platform
>> > > >> >> >> support means for you, which I believe remained
unclear.
>> > > >> >> >> I do not impede the change as you should have
noticed. My
>> > > >> >> >> requirement
>> > > >> >> >> comes from my perception of the support, which
means to me
>> exactly
>> > > >> >> >> two
>> > > >> >> >> things:
>> > > >> >> >> 1. The ability to recognise the code is broken
for the
>> platform
>> > > >> >> >> 2. The ability to test new patches on the platform
>> > > >> >> >> The latter is problematic, as many noticed in
this thread, for
>> > > those
>> > > >> >> >> whose customary environment does not include
Windows.
>> > > >> >> >>
>> > > >> >> >> > If we implemented full "test-patch" support
for Windows on
>> > > trunk,
>> > > >> >> >> > would
>> > > >> >> >> > that
>> > > >> >> >> > fulfill both your items #1 and #2?  Please
note that:
>> > > >> >> >> > a) Pushing the "Patch Available" button
in Jira shall cause
>> a
>> > > >> >> >> > pre-commit
>> > > >> >> >> > build to start within, I believe, 20 minutes.
>> > > >> >> >> > b) That build keeps logs for both java
build and unit tests
>> for
>> > > >> >> >> > several
>> > > >> >> >> > days, that are accessible to all viewers.
>> > > >> >> >>
>> > > >> >> >> In item #1 I mostly asking for the nightly build,
which is
>> simpler
>> > > >> >> >> than "test-patch". The latter would be ideal
from the platform
>> > > >> >> >> support
>> > > >> >> >> viewpoint, but it is for the community to decide
if we want
>> to add
>> > > >> >> >> extra +3 hours to the build.
>> > > >> >> >> Nightly build in my understanding is triggered
by the timer
>> rather
>> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins
build
>> > > >> >> >> configuration
>> > > >> >> >> you can specify it under "Build periodically".
>> > > >> >> >>
>> > > >> >> >> > So, does this provide sufficient on-demand
support that we
>> don't
>> > > >> >> >> > have
>> > > >> >> >> > to
>> > > >> >> >> > implement a whole new on-demand VM support
structure of some
>> > > sort
>> > > >> >> >> > for
>> > > >> >> >> > #2
>> > > >> >> >> > (which would be an extraordinary and impractical
demand)?
>> > > >> >> >>
>> > > >> >> >> I did not mention VMs. Item #2 means a build,
which runs
>> > > >> >> >> "test-patch"
>> > > >> >> >> target with the file specified by a user (instead
of a jira
>> > > >> >> >> attachment).
>> > > >> >> >> When user clicks "Build Now" link a box is displayed
where the
>> > > user
>> > > >> >> >> can enter the file path containing the patch.
This can be
>> > > specified
>> > > >> >> >> in
>> > > >> >> >> the Build Configuration under "This build is
parameterized" by
>> > > >> >> >> choosing AddParameter / FileParameter. The build
can run on
>> the
>> > > same
>> > > >> >> >> Windows machine as the nightly build.
>> > > >> >> >> Such build will let people test their patches
for Windows on
>> > > Jenkins
>> > > >> >> >> if they don't posses a license for the right
version of
>> Windows.
>> > > >> >> >> I hope this will not turn into extraordinary
or impractical
>> > > effort.
>> > > >> >> >>
>> > > >> >> >> Thanks,
>> > > >> >> >> --Konst
>> > > >> >> >>
>> > > >> >> >> > Thanks,
>> > > >> >> >> > --Matt
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin
Shvachko
>> > > >> >> >> > <shv.hadoop@gmail.com>
>> > > >> >> >> > wrote:
>> > > >> >> >> >>
>> > > >> >> >> >> -1
>> > > >> >> >> >> We should have a CI infrastructure
in place before we can
>> > > commit
>> > > >> >> >> >> to
>> > > >> >> >> >> supporting Windows platform.
>> > > >> >> >> >>
>> > > >> >> >> >> Eric is right Win/Cygwin was supported
since day one.
>> > > >> >> >> >> I had a Windows box under my desk running
nightly builds
>> back
>> > > in
>> > > >> >> >> >> 2006-07.
>> > > >> >> >> >> People were irritated but I was filing
windows bugs until
>> 0.22
>> > > >> >> >> >> release.
>> > > >> >> >> >> Times changing and I am glad to see
wider support for Win
>> > > >> >> >> >> platform.
>> > > >> >> >> >>
>> > > >> >> >> >> But in order to make it work you guys
need to put the CI
>> > > process
>> > > >> >> >> >> in
>> > > >> >> >> >> place
>> > > >> >> >> >>
>> > > >> >> >> >> 1. windows jenkins build: could be
nightly or PreCommit.
>> > > >> >> >> >> - Nightly would mean that changes can
be committed to trunk
>> > > based
>> > > >> >> >> >> on
>> > > >> >> >> >> linux PreCommit build. And people will
file bugs if the
>> change
>> > > >> >> >> >> broke
>> > > >> >> >> >> Windows nightly build.
>> > > >> >> >> >> - PreCommit-win build will mean automatic
reporting failed
>> > > tests
>> > > >> >> >> >> to
>> > > >> >> >> >> respective jira blocking commits the
same way as it is now
>> with
>> > > >> >> >> >> linux
>> > > >> >> >> >> PreCommit builds.
>> > > >> >> >> >> We should discuss which way is more
efficient for
>> developers.
>> > > >> >> >> >>
>> > > >> >> >> >> 2. On-demand-windows Jenkins build.
>> > > >> >> >> >> I see it as a build to which I can
attach my patch and the
>> > > build
>> > > >> >> >> >> will
>> > > >> >> >> >> run my changes on a dedicated windows
box.
>> > > >> >> >> >> That way people can test their changes
without having
>> personal
>> > > >> >> >> >> windows
>> > > >> >> >> >> nodes.
>> > > >> >> >> >>
>> > > >> >> >> >> I think this is the minimal set of
requirement for us to be
>> > > able
>> > > >> >> >> >> to
>> > > >> >> >> >> commit to the new platform.
>> > > >> >> >> >> Right now I see only one windows related
build
>> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
>> > > >> >> >> >> Which was failing since Sept 8, 2012
and did not run in the
>> > > last
>> > > >> >> >> >> month.
>> > > >> >> >> >>
>> > > >> >> >> >> Thanks,
>> > > >> >> >> >> --Konst
>> > > >> >> >> >>
>> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric
Baldeschwieler
>> > > >> >> >> >> <eric14@hortonworks.com> wrote:
>> > > >> >> >> >> > +1 (non-binding)
>> > > >> >> >> >> >
>> > > >> >> >> >> > A few of observations:
>> > > >> >> >> >> >
>> > > >> >> >> >> > - Windows has actually been a
supported platform for
>> Hadoop
>> > > >> >> >> >> > since
>> > > >> >> >> >> > 0.1
>> > > >> >> >> >> > .
>> > > >> >> >> >> > Doug championed supporting windows
then and we've
>> continued
>> > > to
>> > > >> >> >> >> > do
>> > > >> >> >> >> > it
>> > > >> >> >> >> > with
>> > > >> >> >> >> > varying vigor over time.  To my
knowledge we've never
>> made a
>> > > >> >> >> >> > decision
>> > > >> >> >> >> > to
>> > > >> >> >> >> > drop windows support.  The change
here is improving our
>> > > support
>> > > >> >> >> >> > and
>> > > >> >> >> >> > dropping
>> > > >> >> >> >> > the requirement of cigwin.  We
had Nutch windows users
>> on the
>> > > >> >> >> >> > list
>> > > >> >> >> >> > in
>> > > >> >> >> >> > 2006
>> > > >> >> >> >> > and we've been supporting windows
FS requirements since
>> > > >> >> >> >> > inception.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A little pragmatism will go
a long way.  As a community
>> > > we've
>> > > >> >> >> >> > got
>> > > >> >> >> >> > to
>> > > >> >> >> >> > stay committed to keeping hadoop
simple (so it does work
>> on
>> > > >> >> >> >> > many
>> > > >> >> >> >> > platforms)
>> > > >> >> >> >> > and extending it to take advantage
of key emerging
>> > > OS/hardware
>> > > >> >> >> >> > features,
>> > > >> >> >> >> > such as containers, new FSs, virtualization,
flash ...
>>  We
>> > > >> >> >> >> > should
>> > > >> >> >> >> > all
>> > > >> >> >> >> > plan
>> > > >> >> >> >> > to let new features & optimizations
emerge that don't
>> work
>> > > >> >> >> >> > everywhere, if
>> > > >> >> >> >> > they are compelling and central
to hadoop's mission of
>> being
>> > > >> >> >> >> > THE
>> > > >> >> >> >> > best
>> > > >> >> >> >> > fabric
>> > > >> >> >> >> > for storing and processing big
data.
>> > > >> >> >> >> >
>> > > >> >> >> >> > - A UI project like KDE has to
deal with the MANY
>> differences
>> > > >> >> >> >> > between
>> > > >> >> >> >> > windows and linux UI APIs.  Hadoop
faces no such complex
>> > > >> >> >> >> > challenge
>> > > >> >> >> >> > and hence
>> > > >> >> >> >> > can be maintained from a single
codeline IMO.  It is
>> mostly
>> > > >> >> >> >> > abstracted from
>> > > >> >> >> >> > the OS APIs via Java and our design
choices.  Where it
>> is not
>> > > >> >> >> >> > we
>> > > >> >> >> >> > can
>> > > >> >> >> >> > continue to add plugable abstractions.
>> > > >> >> >> >> >
>> > > >> >> >> >
>> > > >> >> >> >
>> > > >> >> >
>> > > >> >> >
>> > > >> >
>> > > >> >
>> > > >
>> > > >
>> > >
>>
>
>
>
> --
> Best regards,
>
>    - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)

Mime
View raw message