hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suresh Srinivas <sur...@hortonworks.com>
Subject Re: [Vote] Merge branch-trunk-win to trunk
Date Tue, 26 Mar 2013 00:09:00 GMT
Adding other mailing lists I missed earlier.


Cos,

There is progress being made on that ticket. Also it has nothing to do with
that.

Please follow the discussion here and why this happened due to an invalid
commit that was reverted -
https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=13612650&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13612650

Regards,
Suresh


On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <cos@apache.org> wrote:

> It doesn't look like any progress has been done on the ticket below in the
> last 3 weeks. And now branch-2 can't be compiled because of
>
>
> hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSShell.java:[895,15]
> WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed from
> outside package
>
> That's exactly why I was -1'ing this...
>   Cos
>
> On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > Thanks, gentlemen.  I've opened and taken responsibility for
> > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> agreed
> > to help with the parts that require Jenkins admin access.
> >
> > Thanks,
> > --Matt
> >
> >
> >
> > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> shv.hadoop@gmail.com>wrote:
> >
> > > +1 on the merge.
> > >
> > > I am glad we agreed.
> > > Having Jira to track the CI effort is a good idea.
> > >
> > > Thanks,
> > > --Konstantin
> > >
> > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mfoley@hortonworks.com>
> wrote:
> > > > Thanks.  I agree Windows -1's in test-patch should not block commits.
> > > >
> > > > --Matt
> > > >
> > > >
> > > >
> > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > shv.hadoop@gmail.com>
> > > > wrote:
> > > >>
> > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.com
> >
> > > >> wrote:
> > > >> > Konstantine, you have voted -1, and stated some requirements
> before
> > > >> > you'll
> > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > requirements, I
> > > >> > want to make sure that what I'm proposing will, in fact, satisfy
> you.
> > > >> > That's why I'm asking, if we implement full "test-patch"
> integration
> > > for
> > > >> > Windows, does it seem to you that that would provide adequate
> support?
> > > >>
> > > >> Yes.
> > > >>
> > > >> > I have learned not to presume that my interpretation is correct.
>  My
> > > >> > interpretation of item #1 is that test-patch provides pre-commit
> > > build,
> > > >> > so
> > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > >> > interpreting
> > > >> > it correctly, I simply want your agreement that it would, or
if
> not,
> > > >> > clarification why it won't.
> > > >>
> > > >> I agree it will satisfy my item #1.
> > > >> I did not agree in my previous email, but I changed my mind based
on
> > > >> the latest discussion. I have to explain why now.
> > > >> I was proposing nightly build because I did not want pre-commit
> build
> > > >> for Windows block commits to Linux. But if people are fine just
> ignoring
> > > >> -1s for the Windows part of the build it should be good.
> > > >>
> > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > provides
> > > >> > an
> > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> test,
> > > >> > with
> > > >> > logs available to the developer, so it would satisfy item #2.
 But
> > > >> > rather
> > > >> > than assuming that I am interpreting it correctly, I simply want
> your
> > > >> > agreement that it would, or if not, clarification why it won't.
> > > >>
> > > >> It will satisfy my item #2 in the following way:
> > > >> I can duplicate your pre-commit build for Windows and add an input
> > > >> parameter, which would let people run the build on their patches
> > > >> chosen from local machine rather than attaching them to Jiras.
> > > >>
> > > >> Thanks,
> > > >> --Konstantin
> > > >>
> > > >> > In agile terms, you are the Owner of these requirements.  Please
> give
> > > me
> > > >> > owner feedback as to whether my proposed work sounds like it
will
> > > >> > satisfy
> > > >> > the requirements.
> > > >> >
> > > >> > Thank you,
> > > >> > --Matt
> > > >> >
> > > >> >
> > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > >> > <shv.hadoop@gmail.com>
> > > >> > wrote:
> > > >> >>
> > > >> >> Didn't I explain in details what I am asking for?
> > > >> >>
> > > >> >> Thanks,
> > > >> >> --Konst
> > > >> >>
> > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> mfoley@hortonworks.com>
> > > >> >> wrote:
> > > >> >> > Hi Konstantin,
> > > >> >> > I'd like to point out two things:
> > > >> >> > First, I already committed in this thread (email of
Thu, Feb
> 28,
> > > 2013
> > > >> >> > at
> > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please
stop
> acting
> > > >> >> > like
> > > >> >> > I'm
> > > >> >> > resisting this idea or something.
> > > >> >> > Second, you didn't answer my question, you just kvetched
about
> the
> > > >> >> > phrasing.
> > > >> >> > So I ask again:
> > > >> >> >
> > > >> >> > Will providing full "test-patch" integration (pre-commit
build
> and
> > > >> >> > unit
> > > >> >> > test
> > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> request for
> > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > >> >> >
> > > >> >> > Thanks,
> > > >> >> > --Matt
> > > >> >> >
> > > >> >> >
> > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > >> >> > <shv.hadoop@gmail.com>
> > > >> >> > wrote:
> > > >> >> >>
> > > >> >> >> Hi Matt,
> > > >> >> >>
> > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > mfoley@hortonworks.com>
> > > >> >> >> wrote:
> > > >> >> >> > Konstantin,
> > > >> >> >> > I would like to explore what it would take
to remove this
> > > >> >> >> > perceived
> > > >> >> >> > impediment --
> > > >> >> >>
> > > >> >> >> Glad you decided to explore. Thank you.
> > > >> >> >>
> > > >> >> >> > although I reserve the right to argue that
this is not
> > > >> >> >> > pre-requisite to merging the cross-platform
support patch.
> > > >> >> >>
> > > >> >> >> It's your right indeed. So as mine to question what
the
> platform
> > > >> >> >> support means for you, which I believe remained
unclear.
> > > >> >> >> I do not impede the change as you should have noticed.
My
> > > >> >> >> requirement
> > > >> >> >> comes from my perception of the support, which means
to me
> exactly
> > > >> >> >> two
> > > >> >> >> things:
> > > >> >> >> 1. The ability to recognise the code is broken for
the
> platform
> > > >> >> >> 2. The ability to test new patches on the platform
> > > >> >> >> The latter is problematic, as many noticed in this
thread, for
> > > those
> > > >> >> >> whose customary environment does not include Windows.
> > > >> >> >>
> > > >> >> >> > If we implemented full "test-patch" support
for Windows on
> > > trunk,
> > > >> >> >> > would
> > > >> >> >> > that
> > > >> >> >> > fulfill both your items #1 and #2?  Please
note that:
> > > >> >> >> > a) Pushing the "Patch Available" button in
Jira shall cause
> a
> > > >> >> >> > pre-commit
> > > >> >> >> > build to start within, I believe, 20 minutes.
> > > >> >> >> > b) That build keeps logs for both java build
and unit tests
> for
> > > >> >> >> > several
> > > >> >> >> > days, that are accessible to all viewers.
> > > >> >> >>
> > > >> >> >> In item #1 I mostly asking for the nightly build,
which is
> simpler
> > > >> >> >> than "test-patch". The latter would be ideal from
the platform
> > > >> >> >> support
> > > >> >> >> viewpoint, but it is for the community to decide
if we want
> to add
> > > >> >> >> extra +3 hours to the build.
> > > >> >> >> Nightly build in my understanding is triggered by
the timer
> rather
> > > >> >> >> than by Jira's "submit patch" button.  On Jenkins
build
> > > >> >> >> configuration
> > > >> >> >> you can specify it under "Build periodically".
> > > >> >> >>
> > > >> >> >> > So, does this provide sufficient on-demand
support that we
> don't
> > > >> >> >> > have
> > > >> >> >> > to
> > > >> >> >> > implement a whole new on-demand VM support
structure of some
> > > sort
> > > >> >> >> > for
> > > >> >> >> > #2
> > > >> >> >> > (which would be an extraordinary and impractical
demand)?
> > > >> >> >>
> > > >> >> >> I did not mention VMs. Item #2 means a build, which
runs
> > > >> >> >> "test-patch"
> > > >> >> >> target with the file specified by a user (instead
of a jira
> > > >> >> >> attachment).
> > > >> >> >> When user clicks "Build Now" link a box is displayed
where the
> > > user
> > > >> >> >> can enter the file path containing the patch. This
can be
> > > specified
> > > >> >> >> in
> > > >> >> >> the Build Configuration under "This build is parameterized"
by
> > > >> >> >> choosing AddParameter / FileParameter. The build
can run on
> the
> > > same
> > > >> >> >> Windows machine as the nightly build.
> > > >> >> >> Such build will let people test their patches for
Windows on
> > > Jenkins
> > > >> >> >> if they don't posses a license for the right version
of
> Windows.
> > > >> >> >> I hope this will not turn into extraordinary or
impractical
> > > effort.
> > > >> >> >>
> > > >> >> >> Thanks,
> > > >> >> >> --Konst
> > > >> >> >>
> > > >> >> >> > Thanks,
> > > >> >> >> > --Matt
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin
Shvachko
> > > >> >> >> > <shv.hadoop@gmail.com>
> > > >> >> >> > wrote:
> > > >> >> >> >>
> > > >> >> >> >> -1
> > > >> >> >> >> We should have a CI infrastructure in place
before we can
> > > commit
> > > >> >> >> >> to
> > > >> >> >> >> supporting Windows platform.
> > > >> >> >> >>
> > > >> >> >> >> Eric is right Win/Cygwin was supported
since day one.
> > > >> >> >> >> I had a Windows box under my desk running
nightly builds
> back
> > > in
> > > >> >> >> >> 2006-07.
> > > >> >> >> >> People were irritated but I was filing
windows bugs until
> 0.22
> > > >> >> >> >> release.
> > > >> >> >> >> Times changing and I am glad to see wider
support for Win
> > > >> >> >> >> platform.
> > > >> >> >> >>
> > > >> >> >> >> But in order to make it work you guys need
to put the CI
> > > process
> > > >> >> >> >> in
> > > >> >> >> >> place
> > > >> >> >> >>
> > > >> >> >> >> 1. windows jenkins build: could be nightly
or PreCommit.
> > > >> >> >> >> - Nightly would mean that changes can be
committed to trunk
> > > based
> > > >> >> >> >> on
> > > >> >> >> >> linux PreCommit build. And people will
file bugs if the
> change
> > > >> >> >> >> broke
> > > >> >> >> >> Windows nightly build.
> > > >> >> >> >> - PreCommit-win build will mean automatic
reporting failed
> > > tests
> > > >> >> >> >> to
> > > >> >> >> >> respective jira blocking commits the same
way as it is now
> with
> > > >> >> >> >> linux
> > > >> >> >> >> PreCommit builds.
> > > >> >> >> >> We should discuss which way is more efficient
for
> developers.
> > > >> >> >> >>
> > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > >> >> >> >> I see it as a build to which I can attach
my patch and the
> > > build
> > > >> >> >> >> will
> > > >> >> >> >> run my changes on a dedicated windows box.
> > > >> >> >> >> That way people can test their changes
without having
> personal
> > > >> >> >> >> windows
> > > >> >> >> >> nodes.
> > > >> >> >> >>
> > > >> >> >> >> I think this is the minimal set of requirement
for us to be
> > > able
> > > >> >> >> >> to
> > > >> >> >> >> commit to the new platform.
> > > >> >> >> >> Right now I see only one windows related
build
> > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > >> >> >> >> Which was failing since Sept 8, 2012 and
did not run in the
> > > last
> > > >> >> >> >> month.
> > > >> >> >> >>
> > > >> >> >> >> Thanks,
> > > >> >> >> >> --Konst
> > > >> >> >> >>
> > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > >> >> >> >> <eric14@hortonworks.com> wrote:
> > > >> >> >> >> > +1 (non-binding)
> > > >> >> >> >> >
> > > >> >> >> >> > A few of observations:
> > > >> >> >> >> >
> > > >> >> >> >> > - Windows has actually been a supported
platform for
> Hadoop
> > > >> >> >> >> > since
> > > >> >> >> >> > 0.1
> > > >> >> >> >> > .
> > > >> >> >> >> > Doug championed supporting windows
then and we've
> continued
> > > to
> > > >> >> >> >> > do
> > > >> >> >> >> > it
> > > >> >> >> >> > with
> > > >> >> >> >> > varying vigor over time.  To my knowledge
we've never
> made a
> > > >> >> >> >> > decision
> > > >> >> >> >> > to
> > > >> >> >> >> > drop windows support.  The change
here is improving our
> > > support
> > > >> >> >> >> > and
> > > >> >> >> >> > dropping
> > > >> >> >> >> > the requirement of cigwin.  We had
Nutch windows users
> on the
> > > >> >> >> >> > list
> > > >> >> >> >> > in
> > > >> >> >> >> > 2006
> > > >> >> >> >> > and we've been supporting windows
FS requirements since
> > > >> >> >> >> > inception.
> > > >> >> >> >> >
> > > >> >> >> >> > - A little pragmatism will go a long
way.  As a community
> > > we've
> > > >> >> >> >> > got
> > > >> >> >> >> > to
> > > >> >> >> >> > stay committed to keeping hadoop simple
(so it does work
> on
> > > >> >> >> >> > many
> > > >> >> >> >> > platforms)
> > > >> >> >> >> > and extending it to take advantage
of key emerging
> > > OS/hardware
> > > >> >> >> >> > features,
> > > >> >> >> >> > such as containers, new FSs, virtualization,
flash ...
>  We
> > > >> >> >> >> > should
> > > >> >> >> >> > all
> > > >> >> >> >> > plan
> > > >> >> >> >> > to let new features & optimizations
emerge that don't
> work
> > > >> >> >> >> > everywhere, if
> > > >> >> >> >> > they are compelling and central to
hadoop's mission of
> being
> > > >> >> >> >> > THE
> > > >> >> >> >> > best
> > > >> >> >> >> > fabric
> > > >> >> >> >> > for storing and processing big data.
> > > >> >> >> >> >
> > > >> >> >> >> > - A UI project like KDE has to deal
with the MANY
> differences
> > > >> >> >> >> > between
> > > >> >> >> >> > windows and linux UI APIs.  Hadoop
faces no such complex
> > > >> >> >> >> > challenge
> > > >> >> >> >> > and hence
> > > >> >> >> >> > can be maintained from a single codeline
IMO.  It is
> mostly
> > > >> >> >> >> > abstracted from
> > > >> >> >> >> > the OS APIs via Java and our design
choices.  Where it
> is not
> > > >> >> >> >> > we
> > > >> >> >> >> > can
> > > >> >> >> >> > continue to add plugable abstractions.
> > > >> >> >> >> >
> > > >> >> >> >
> > > >> >> >> >
> > > >> >> >
> > > >> >> >
> > > >> >
> > > >> >
> > > >
> > > >
> > >
>



-- 
http://hortonworks.com/download/

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message