hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Purtell <apurt...@apache.org>
Subject Re: [Vote] Merge branch-trunk-win to trunk
Date Mon, 25 Mar 2013 21:53:32 GMT
Sorry, that was my error selecting the wrong reply option.


On Mon, Mar 25, 2013 at 10:25 PM, Konstantin Shvachko
<shv.hadoop@gmail.com>wrote:

> Andrew, this used to be on all -dev lists. Let's keep it that way.
>
> To the point.
> Does this mean that people are silently porting windows changes to
> branch-2?
> New features on a branch should be voted first, no?
>
> Thanks,
> --Konstantin
>
>
> On Mon, Mar 25, 2013 at 1:36 PM, Andrew Purtell <apurtell@apache.org>
> wrote:
> > Noticed this too. Simply a 'public' modifier is missing, but it's unclear
> > how this could not have been caught prior to check-in.
> >
> >
> > On Mon, Mar 25, 2013 at 9:17 PM, Konstantin Boudnik <cos@apache.org>
> wrote:
> >
> >> It doesn't look like any progress has been done on the ticket below in
> the
> >> last 3 weeks. And now branch-2 can't be compiled because of
> >>
> >>
> >>
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> >> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed
> from
> >> outside package
> >>
> >> That's exactly why I was -1'ing this...
> >>   Cos
> >>
> >> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> >> > Thanks, gentlemen.  I've opened and taken responsibility for
> >> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> >> agreed
> >> > to help with the parts that require Jenkins admin access.
> >> >
> >> > Thanks,
> >> > --Matt
> >> >
> >> >
> >> >
> >> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> >> shv.hadoop@gmail.com>wrote:
> >> >
> >> > > +1 on the merge.
> >> > >
> >> > > I am glad we agreed.
> >> > > Having Jira to track the CI effort is a good idea.
> >> > >
> >> > > Thanks,
> >> > > --Konstantin
> >> > >
> >> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mfoley@hortonworks.com>
> >> wrote:
> >> > > > Thanks.  I agree Windows -1's in test-patch should not block
> commits.
> >> > > >
> >> > > > --Matt
> >> > > >
> >> > > >
> >> > > >
> >> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> >> > > shv.hadoop@gmail.com>
> >> > > > wrote:
> >> > > >>
> >> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <
> mfoley@hortonworks.com
> >> >
> >> > > >> wrote:
> >> > > >> > Konstantine, you have voted -1, and stated some requirements
> >> before
> >> > > >> > you'll
> >> > > >> > withdraw that -1.  As I plan to do work to fulfill those
> >> > > requirements, I
> >> > > >> > want to make sure that what I'm proposing will, in fact,
> satisfy
> >> you.
> >> > > >> > That's why I'm asking, if we implement full "test-patch"
> >> integration
> >> > > for
> >> > > >> > Windows, does it seem to you that that would provide
adequate
> >> support?
> >> > > >>
> >> > > >> Yes.
> >> > > >>
> >> > > >> > I have learned not to presume that my interpretation
is
> correct.
> >>  My
> >> > > >> > interpretation of item #1 is that test-patch provides
> pre-commit
> >> > > build,
> >> > > >> > so
> >> > > >> > it would satisfy item #1.  But rather than assuming
that I am
> >> > > >> > interpreting
> >> > > >> > it correctly, I simply want your agreement that it would,
or if
> >> not,
> >> > > >> > clarification why it won't.
> >> > > >>
> >> > > >> I agree it will satisfy my item #1.
> >> > > >> I did not agree in my previous email, but I changed my mind
> based on
> >> > > >> the latest discussion. I have to explain why now.
> >> > > >> I was proposing nightly build because I did not want pre-commit
> >> build
> >> > > >> for Windows block commits to Linux. But if people are fine
just
> >> ignoring
> >> > > >> -1s for the Windows part of the build it should be good.
> >> > > >>
> >> > > >> > Regarding item #2, it is also my interpretation that
test-patch
> >> > > provides
> >> > > >> > an
> >> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build
and unit
> >> test,
> >> > > >> > with
> >> > > >> > logs available to the developer, so it would satisfy
item #2.
>  But
> >> > > >> > rather
> >> > > >> > than assuming that I am interpreting it correctly, I
simply
> want
> >> your
> >> > > >> > agreement that it would, or if not, clarification why
it won't.
> >> > > >>
> >> > > >> It will satisfy my item #2 in the following way:
> >> > > >> I can duplicate your pre-commit build for Windows and add
an
> input
> >> > > >> parameter, which would let people run the build on their
patches
> >> > > >> chosen from local machine rather than attaching them to Jiras.
> >> > > >>
> >> > > >> Thanks,
> >> > > >> --Konstantin
> >> > > >>
> >> > > >> > In agile terms, you are the Owner of these requirements.
>  Please
> >> give
> >> > > me
> >> > > >> > owner feedback as to whether my proposed work sounds
like it
> will
> >> > > >> > satisfy
> >> > > >> > the requirements.
> >> > > >> >
> >> > > >> > Thank you,
> >> > > >> > --Matt
> >> > > >> >
> >> > > >> >
> >> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> >> > > >> > <shv.hadoop@gmail.com>
> >> > > >> > wrote:
> >> > > >> >>
> >> > > >> >> Didn't I explain in details what I am asking for?
> >> > > >> >>
> >> > > >> >> Thanks,
> >> > > >> >> --Konst
> >> > > >> >>
> >> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> >> mfoley@hortonworks.com>
> >> > > >> >> wrote:
> >> > > >> >> > Hi Konstantin,
> >> > > >> >> > I'd like to point out two things:
> >> > > >> >> > First, I already committed in this thread (email
of Thu, Feb
> >> 28,
> >> > > 2013
> >> > > >> >> > at
> >> > > >> >> > 6:01 PM) to providing CI for Windows builds.
 So please stop
> >> acting
> >> > > >> >> > like
> >> > > >> >> > I'm
> >> > > >> >> > resisting this idea or something.
> >> > > >> >> > Second, you didn't answer my question, you
just kvetched
> about
> >> the
> >> > > >> >> > phrasing.
> >> > > >> >> > So I ask again:
> >> > > >> >> >
> >> > > >> >> > Will providing full "test-patch" integration
(pre-commit
> build
> >> and
> >> > > >> >> > unit
> >> > > >> >> > test
> >> > > >> >> > triggered by Jira "Patch Available" state)
satisfy your
> >> request for
> >> > > >> >> > functionality #1 and #2?  Yes or no, please.
> >> > > >> >> >
> >> > > >> >> > Thanks,
> >> > > >> >> > --Matt
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin
Shvachko
> >> > > >> >> > <shv.hadoop@gmail.com>
> >> > > >> >> > wrote:
> >> > > >> >> >>
> >> > > >> >> >> Hi Matt,
> >> > > >> >> >>
> >> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley
<
> >> > > mfoley@hortonworks.com>
> >> > > >> >> >> wrote:
> >> > > >> >> >> > Konstantin,
> >> > > >> >> >> > I would like to explore what it would
take to remove this
> >> > > >> >> >> > perceived
> >> > > >> >> >> > impediment --
> >> > > >> >> >>
> >> > > >> >> >> Glad you decided to explore. Thank you.
> >> > > >> >> >>
> >> > > >> >> >> > although I reserve the right to argue
that this is not
> >> > > >> >> >> > pre-requisite to merging the cross-platform
support
> patch.
> >> > > >> >> >>
> >> > > >> >> >> It's your right indeed. So as mine to question
what the
> >> platform
> >> > > >> >> >> support means for you, which I believe
remained unclear.
> >> > > >> >> >> I do not impede the change as you should
have noticed. My
> >> > > >> >> >> requirement
> >> > > >> >> >> comes from my perception of the support,
which means to me
> >> exactly
> >> > > >> >> >> two
> >> > > >> >> >> things:
> >> > > >> >> >> 1. The ability to recognise the code is
broken for the
> >> platform
> >> > > >> >> >> 2. The ability to test new patches on the
platform
> >> > > >> >> >> The latter is problematic, as many noticed
in this thread,
> for
> >> > > those
> >> > > >> >> >> whose customary environment does not include
Windows.
> >> > > >> >> >>
> >> > > >> >> >> > If we implemented full "test-patch"
support for Windows
> on
> >> > > trunk,
> >> > > >> >> >> > would
> >> > > >> >> >> > that
> >> > > >> >> >> > fulfill both your items #1 and #2?
 Please note that:
> >> > > >> >> >> > a) Pushing the "Patch Available" button
in Jira shall
> cause
> >> a
> >> > > >> >> >> > pre-commit
> >> > > >> >> >> > build to start within, I believe,
20 minutes.
> >> > > >> >> >> > b) That build keeps logs for both
java build and unit
> tests
> >> for
> >> > > >> >> >> > several
> >> > > >> >> >> > days, that are accessible to all viewers.
> >> > > >> >> >>
> >> > > >> >> >> In item #1 I mostly asking for the nightly
build, which is
> >> simpler
> >> > > >> >> >> than "test-patch". The latter would be
ideal from the
> platform
> >> > > >> >> >> support
> >> > > >> >> >> viewpoint, but it is for the community
to decide if we want
> >> to add
> >> > > >> >> >> extra +3 hours to the build.
> >> > > >> >> >> Nightly build in my understanding is triggered
by the timer
> >> rather
> >> > > >> >> >> than by Jira's "submit patch" button. 
On Jenkins build
> >> > > >> >> >> configuration
> >> > > >> >> >> you can specify it under "Build periodically".
> >> > > >> >> >>
> >> > > >> >> >> > So, does this provide sufficient on-demand
support that
> we
> >> don't
> >> > > >> >> >> > have
> >> > > >> >> >> > to
> >> > > >> >> >> > implement a whole new on-demand VM
support structure of
> some
> >> > > sort
> >> > > >> >> >> > for
> >> > > >> >> >> > #2
> >> > > >> >> >> > (which would be an extraordinary and
impractical demand)?
> >> > > >> >> >>
> >> > > >> >> >> I did not mention VMs. Item #2 means a
build, which runs
> >> > > >> >> >> "test-patch"
> >> > > >> >> >> target with the file specified by a user
(instead of a jira
> >> > > >> >> >> attachment).
> >> > > >> >> >> When user clicks "Build Now" link a box
is displayed where
> the
> >> > > user
> >> > > >> >> >> can enter the file path containing the
patch. This can be
> >> > > specified
> >> > > >> >> >> in
> >> > > >> >> >> the Build Configuration under "This build
is
> parameterized" by
> >> > > >> >> >> choosing AddParameter / FileParameter.
The build can run on
> >> the
> >> > > same
> >> > > >> >> >> Windows machine as the nightly build.
> >> > > >> >> >> Such build will let people test their patches
for Windows
> on
> >> > > Jenkins
> >> > > >> >> >> if they don't posses a license for the
right version of
> >> Windows.
> >> > > >> >> >> I hope this will not turn into extraordinary
or impractical
> >> > > effort.
> >> > > >> >> >>
> >> > > >> >> >> Thanks,
> >> > > >> >> >> --Konst
> >> > > >> >> >>
> >> > > >> >> >> > Thanks,
> >> > > >> >> >> > --Matt
> >> > > >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin
Shvachko
> >> > > >> >> >> > <shv.hadoop@gmail.com>
> >> > > >> >> >> > wrote:
> >> > > >> >> >> >>
> >> > > >> >> >> >> -1
> >> > > >> >> >> >> We should have a CI infrastructure
in place before we
> can
> >> > > commit
> >> > > >> >> >> >> to
> >> > > >> >> >> >> supporting Windows platform.
> >> > > >> >> >> >>
> >> > > >> >> >> >> Eric is right Win/Cygwin was supported
since day one.
> >> > > >> >> >> >> I had a Windows box under my desk
running nightly builds
> >> back
> >> > > in
> >> > > >> >> >> >> 2006-07.
> >> > > >> >> >> >> People were irritated but I was
filing windows bugs
> until
> >> 0.22
> >> > > >> >> >> >> release.
> >> > > >> >> >> >> Times changing and I am glad to
see wider support for
> Win
> >> > > >> >> >> >> platform.
> >> > > >> >> >> >>
> >> > > >> >> >> >> But in order to make it work you
guys need to put the CI
> >> > > process
> >> > > >> >> >> >> in
> >> > > >> >> >> >> place
> >> > > >> >> >> >>
> >> > > >> >> >> >> 1. windows jenkins build: could
be nightly or PreCommit.
> >> > > >> >> >> >> - Nightly would mean that changes
can be committed to
> trunk
> >> > > based
> >> > > >> >> >> >> on
> >> > > >> >> >> >> linux PreCommit build. And people
will file bugs if the
> >> change
> >> > > >> >> >> >> broke
> >> > > >> >> >> >> Windows nightly build.
> >> > > >> >> >> >> - PreCommit-win build will mean
automatic reporting
> failed
> >> > > tests
> >> > > >> >> >> >> to
> >> > > >> >> >> >> respective jira blocking commits
the same way as it is
> now
> >> with
> >> > > >> >> >> >> linux
> >> > > >> >> >> >> PreCommit builds.
> >> > > >> >> >> >> We should discuss which way is
more efficient for
> >> developers.
> >> > > >> >> >> >>
> >> > > >> >> >> >> 2. On-demand-windows Jenkins build.
> >> > > >> >> >> >> I see it as a build to which I
can attach my patch and
> the
> >> > > build
> >> > > >> >> >> >> will
> >> > > >> >> >> >> run my changes on a dedicated
windows box.
> >> > > >> >> >> >> That way people can test their
changes without having
> >> personal
> >> > > >> >> >> >> windows
> >> > > >> >> >> >> nodes.
> >> > > >> >> >> >>
> >> > > >> >> >> >> I think this is the minimal set
of requirement for us
> to be
> >> > > able
> >> > > >> >> >> >> to
> >> > > >> >> >> >> commit to the new platform.
> >> > > >> >> >> >> Right now I see only one windows
related build
> >> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> >> > > >> >> >> >> Which was failing since Sept 8,
2012 and did not run in
> the
> >> > > last
> >> > > >> >> >> >> month.
> >> > > >> >> >> >>
> >> > > >> >> >> >> Thanks,
> >> > > >> >> >> >> --Konst
> >> > > >> >> >> >>
> >> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM,
Eric Baldeschwieler
> >> > > >> >> >> >> <eric14@hortonworks.com>
wrote:
> >> > > >> >> >> >> > +1 (non-binding)
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > A few of observations:
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - Windows has actually been
a supported platform for
> >> Hadoop
> >> > > >> >> >> >> > since
> >> > > >> >> >> >> > 0.1
> >> > > >> >> >> >> > .
> >> > > >> >> >> >> > Doug championed supporting
windows then and we've
> >> continued
> >> > > to
> >> > > >> >> >> >> > do
> >> > > >> >> >> >> > it
> >> > > >> >> >> >> > with
> >> > > >> >> >> >> > varying vigor over time.
 To my knowledge we've never
> >> made a
> >> > > >> >> >> >> > decision
> >> > > >> >> >> >> > to
> >> > > >> >> >> >> > drop windows support.  The
change here is improving
> our
> >> > > support
> >> > > >> >> >> >> > and
> >> > > >> >> >> >> > dropping
> >> > > >> >> >> >> > the requirement of cigwin.
 We had Nutch windows users
> >> on the
> >> > > >> >> >> >> > list
> >> > > >> >> >> >> > in
> >> > > >> >> >> >> > 2006
> >> > > >> >> >> >> > and we've been supporting
windows FS requirements
> since
> >> > > >> >> >> >> > inception.
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - A little pragmatism will
go a long way.  As a
> community
> >> > > we've
> >> > > >> >> >> >> > got
> >> > > >> >> >> >> > to
> >> > > >> >> >> >> > stay committed to keeping
hadoop simple (so it does
> work
> >> on
> >> > > >> >> >> >> > many
> >> > > >> >> >> >> > platforms)
> >> > > >> >> >> >> > and extending it to take
advantage of key emerging
> >> > > OS/hardware
> >> > > >> >> >> >> > features,
> >> > > >> >> >> >> > such as containers, new FSs,
virtualization, flash ...
> >>  We
> >> > > >> >> >> >> > should
> >> > > >> >> >> >> > all
> >> > > >> >> >> >> > plan
> >> > > >> >> >> >> > to let new features &
optimizations emerge that don't
> >> work
> >> > > >> >> >> >> > everywhere, if
> >> > > >> >> >> >> > they are compelling and central
to hadoop's mission of
> >> being
> >> > > >> >> >> >> > THE
> >> > > >> >> >> >> > best
> >> > > >> >> >> >> > fabric
> >> > > >> >> >> >> > for storing and processing
big data.
> >> > > >> >> >> >> >
> >> > > >> >> >> >> > - A UI project like KDE has
to deal with the MANY
> >> differences
> >> > > >> >> >> >> > between
> >> > > >> >> >> >> > windows and linux UI APIs.
 Hadoop faces no such
> complex
> >> > > >> >> >> >> > challenge
> >> > > >> >> >> >> > and hence
> >> > > >> >> >> >> > can be maintained from a
single codeline IMO.  It is
> >> mostly
> >> > > >> >> >> >> > abstracted from
> >> > > >> >> >> >> > the OS APIs via Java and
our design choices.  Where it
> >> is not
> >> > > >> >> >> >> > we
> >> > > >> >> >> >> > can
> >> > > >> >> >> >> > continue to add plugable
abstractions.
> >> > > >> >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >> >
> >> > > >> >> >
> >> > > >> >> >
> >> > > >> >
> >> > > >> >
> >> > > >
> >> > > >
> >> > >
> >>
> >
> >
> >
> > --
> > Best regards,
> >
> >    - Andy
> >
> > Problems worthy of attack prove their worth by hitting back. - Piet Hein
> > (via Tom White)
>



-- 
Best regards,

   - Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message