Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
Date: Mon, 25 Mar 2013 19:14:04 -0700
From: Konstantin Boudnik <cos@apache.org>
To: hdfs-dev@hadoop.apache.org
Cc: "mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
        Konstantin Shvachko <shv.hadoop@gmail.com>,
        "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>,
        "yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
Subject: Re: [Vote] Merge branch-trunk-win to trunk
Message-ID: <20130326021404.GB3258@tpx>
Mail-Followup-To: hdfs-dev@hadoop.apache.org,
	"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>,
	Konstantin Shvachko <shv.hadoop@gmail.com>,
	"common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>,
	"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
References: 
 <CAKtuutGc1VKovv_JpRCjC7xx989G-N=0JdSSTd2XxD==Y1Kshg@mail.gmail.com>
 <CA+z3+9EFgvEXH+0xNBTL_TK40+FZAtG+SaZ0GMj+=JsBRUf7=A@mail.gmail.com>
 <CAKtuutHO_P0cesJNednhPwNK=_zeYkB2xa3MyoU3X3=Fztdeng@mail.gmail.com>
 <CA+z3+9H1uw_C-=dWmcEk_p6Yi6S8oqorKHa+F1Wg0W40ReF_pg@mail.gmail.com>
 <CAKtuutG=86QL11dPySFwFd=XU_hE6dxJZ0a0SuZS=-t9ZwPKPw@mail.gmail.com>
 <CA+z3+9FXB2rxs+q2JTERVBiuUPHdvdGcSqJVYSqVif+Qq1GvxA@mail.gmail.com>
 <CAKtuutHdE1xOLOGGC0mz_aoKvhpf1oPwn8Lmc9M-9NhbOkmhBg@mail.gmail.com>
 <CA+z3+9GuGn+wb7yoCNL7P8urNNz0C6piadfHw2RuN=fB8X5ORQ@mail.gmail.com>
 <20130325201742.GG1612@linspire.com>
 <CADdVdVHvZsDqvVT+170KRs=T9SvbaA1Jx=wWNr8=Jfem6urgBA@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256;
	protocol="application/pgp-signature"; boundary="IrhDeMKUP4DT/M7F"
Content-Disposition: inline
In-Reply-To: 
 <CADdVdVHvZsDqvVT+170KRs=T9SvbaA1Jx=wWNr8=Jfem6urgBA@mail.gmail.com>
User-Agent: Mutt/1.5.21 (2010-09-15)

--IrhDeMKUP4DT/M7F
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Yes, you are right of course - the mis-merged commit is the cause. Thanks f=
or
pointing this out!

I think it would be beneficial if we had branch-2 on going build in the
Jenkins. I will go ahead and create one tonight.

Cos

On Mon, Mar 25, 2013 at 05:09PM, Suresh Srinivas wrote:
> Adding other mailing lists I missed earlier.
>=20
> Cos,
>=20
> There is progress being made on that ticket. Also it has nothing to do wi=
th
> that.
>=20
> Please follow the discussion here and why this happened due to an invalid
> commit that was reverted -
> https://issues.apache.org/jira/browse/HDFS-4615?focusedCommentId=3D136126=
50&page=3Dcom.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#=
comment-13612650
>=20
> Regards,
> Suresh
>=20
>=20
> On Mon, Mar 25, 2013 at 1:17 PM, Konstantin Boudnik <cos@apache.org> wrot=
e:
>=20
> > It doesn't look like any progress has been done on the ticket below in =
the
> > last 3 weeks. And now branch-2 can't be compiled because of
> >
> >
> > hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/Te=
stDFSShell.java:[895,15]
> > WINDOWS is not public in org.apache.hadoop.fs.Path; cannot be accessed =
=66rom
> > outside package
> >
> > That's exactly why I was -1'ing this...
> >   Cos
> >
> > On Mon, Mar 04, 2013 at 05:41PM, Matt Foley wrote:
> > > Thanks, gentlemen.  I've opened and taken responsibility for
> > > https://issues.apache.org/jira/browse/HADOOP-9359.  Giri Kesavan has
> > agreed
> > > to help with the parts that require Jenkins admin access.
> > >
> > > Thanks,
> > > --Matt
> > >
> > >
> > >
> > > On Mon, Mar 4, 2013 at 5:00 PM, Konstantin Shvachko <
> > shv.hadoop@gmail.com>wrote:
> > >
> > > > +1 on the merge.
> > > >
> > > > I am glad we agreed.
> > > > Having Jira to track the CI effort is a good idea.
> > > >
> > > > Thanks,
> > > > --Konstantin
> > > >
> > > > On Mon, Mar 4, 2013 at 3:29 PM, Matt Foley <mfoley@hortonworks.com>
> > wrote:
> > > > > Thanks.  I agree Windows -1's in test-patch should not block comm=
its.
> > > > >
> > > > > --Matt
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Mar 4, 2013 at 2:30 PM, Konstantin Shvachko <
> > > > shv.hadoop@gmail.com>
> > > > > wrote:
> > > > >>
> > > > >> On Mon, Mar 4, 2013 at 12:22 PM, Matt Foley <mfoley@hortonworks.=
com
> > >
> > > > >> wrote:
> > > > >> > Konstantine, you have voted -1, and stated some requirements
> > before
> > > > >> > you'll
> > > > >> > withdraw that -1.  As I plan to do work to fulfill those
> > > > requirements, I
> > > > >> > want to make sure that what I'm proposing will, in fact, satis=
fy
> > you.
> > > > >> > That's why I'm asking, if we implement full "test-patch"
> > integration
> > > > for
> > > > >> > Windows, does it seem to you that that would provide adequate
> > support?
> > > > >>
> > > > >> Yes.
> > > > >>
> > > > >> > I have learned not to presume that my interpretation is correc=
t.
> >  My
> > > > >> > interpretation of item #1 is that test-patch provides pre-comm=
it
> > > > build,
> > > > >> > so
> > > > >> > it would satisfy item #1.  But rather than assuming that I am
> > > > >> > interpreting
> > > > >> > it correctly, I simply want your agreement that it would, or if
> > not,
> > > > >> > clarification why it won't.
> > > > >>
> > > > >> I agree it will satisfy my item #1.
> > > > >> I did not agree in my previous email, but I changed my mind base=
d on
> > > > >> the latest discussion. I have to explain why now.
> > > > >> I was proposing nightly build because I did not want pre-commit
> > build
> > > > >> for Windows block commits to Linux. But if people are fine just
> > ignoring
> > > > >> -1s for the Windows part of the build it should be good.
> > > > >>
> > > > >> > Regarding item #2, it is also my interpretation that test-patch
> > > > provides
> > > > >> > an
> > > > >> > on-demand (perhaps 20-minutes deferred) Jenkins build and unit
> > test,
> > > > >> > with
> > > > >> > logs available to the developer, so it would satisfy item #2. =
 But
> > > > >> > rather
> > > > >> > than assuming that I am interpreting it correctly, I simply wa=
nt
> > your
> > > > >> > agreement that it would, or if not, clarification why it won't.
> > > > >>
> > > > >> It will satisfy my item #2 in the following way:
> > > > >> I can duplicate your pre-commit build for Windows and add an inp=
ut
> > > > >> parameter, which would let people run the build on their patches
> > > > >> chosen from local machine rather than attaching them to Jiras.
> > > > >>
> > > > >> Thanks,
> > > > >> --Konstantin
> > > > >>
> > > > >> > In agile terms, you are the Owner of these requirements.  Plea=
se
> > give
> > > > me
> > > > >> > owner feedback as to whether my proposed work sounds like it w=
ill
> > > > >> > satisfy
> > > > >> > the requirements.
> > > > >> >
> > > > >> > Thank you,
> > > > >> > --Matt
> > > > >> >
> > > > >> >
> > > > >> > On Sun, Mar 3, 2013 at 12:16 PM, Konstantin Shvachko
> > > > >> > <shv.hadoop@gmail.com>
> > > > >> > wrote:
> > > > >> >>
> > > > >> >> Didn't I explain in details what I am asking for?
> > > > >> >>
> > > > >> >> Thanks,
> > > > >> >> --Konst
> > > > >> >>
> > > > >> >> On Sun, Mar 3, 2013 at 11:08 AM, Matt Foley <
> > mfoley@hortonworks.com>
> > > > >> >> wrote:
> > > > >> >> > Hi Konstantin,
> > > > >> >> > I'd like to point out two things:
> > > > >> >> > First, I already committed in this thread (email of Thu, Feb
> > 28,
> > > > 2013
> > > > >> >> > at
> > > > >> >> > 6:01 PM) to providing CI for Windows builds.  So please stop
> > acting
> > > > >> >> > like
> > > > >> >> > I'm
> > > > >> >> > resisting this idea or something.
> > > > >> >> > Second, you didn't answer my question, you just kvetched ab=
out
> > the
> > > > >> >> > phrasing.
> > > > >> >> > So I ask again:
> > > > >> >> >
> > > > >> >> > Will providing full "test-patch" integration (pre-commit bu=
ild
> > and
> > > > >> >> > unit
> > > > >> >> > test
> > > > >> >> > triggered by Jira "Patch Available" state) satisfy your
> > request for
> > > > >> >> > functionality #1 and #2?  Yes or no, please.
> > > > >> >> >
> > > > >> >> > Thanks,
> > > > >> >> > --Matt
> > > > >> >> >
> > > > >> >> >
> > > > >> >> > On Sat, Mar 2, 2013 at 7:32 PM, Konstantin Shvachko
> > > > >> >> > <shv.hadoop@gmail.com>
> > > > >> >> > wrote:
> > > > >> >> >>
> > > > >> >> >> Hi Matt,
> > > > >> >> >>
> > > > >> >> >> On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley <
> > > > mfoley@hortonworks.com>
> > > > >> >> >> wrote:
> > > > >> >> >> > Konstantin,
> > > > >> >> >> > I would like to explore what it would take to remove this
> > > > >> >> >> > perceived
> > > > >> >> >> > impediment --
> > > > >> >> >>
> > > > >> >> >> Glad you decided to explore. Thank you.
> > > > >> >> >>
> > > > >> >> >> > although I reserve the right to argue that this is not
> > > > >> >> >> > pre-requisite to merging the cross-platform support patc=
h.
> > > > >> >> >>
> > > > >> >> >> It's your right indeed. So as mine to question what the
> > platform
> > > > >> >> >> support means for you, which I believe remained unclear.
> > > > >> >> >> I do not impede the change as you should have noticed. My
> > > > >> >> >> requirement
> > > > >> >> >> comes from my perception of the support, which means to me
> > exactly
> > > > >> >> >> two
> > > > >> >> >> things:
> > > > >> >> >> 1. The ability to recognise the code is broken for the
> > platform
> > > > >> >> >> 2. The ability to test new patches on the platform
> > > > >> >> >> The latter is problematic, as many noticed in this thread,=
 for
> > > > those
> > > > >> >> >> whose customary environment does not include Windows.
> > > > >> >> >>
> > > > >> >> >> > If we implemented full "test-patch" support for Windows =
on
> > > > trunk,
> > > > >> >> >> > would
> > > > >> >> >> > that
> > > > >> >> >> > fulfill both your items #1 and #2?  Please note that:
> > > > >> >> >> > a) Pushing the "Patch Available" button in Jira shall ca=
use
> > a
> > > > >> >> >> > pre-commit
> > > > >> >> >> > build to start within, I believe, 20 minutes.
> > > > >> >> >> > b) That build keeps logs for both java build and unit te=
sts
> > for
> > > > >> >> >> > several
> > > > >> >> >> > days, that are accessible to all viewers.
> > > > >> >> >>
> > > > >> >> >> In item #1 I mostly asking for the nightly build, which is
> > simpler
> > > > >> >> >> than "test-patch". The latter would be ideal from the plat=
form
> > > > >> >> >> support
> > > > >> >> >> viewpoint, but it is for the community to decide if we want
> > to add
> > > > >> >> >> extra +3 hours to the build.
> > > > >> >> >> Nightly build in my understanding is triggered by the timer
> > rather
> > > > >> >> >> than by Jira's "submit patch" button.  On Jenkins build
> > > > >> >> >> configuration
> > > > >> >> >> you can specify it under "Build periodically".
> > > > >> >> >>
> > > > >> >> >> > So, does this provide sufficient on-demand support that =
we
> > don't
> > > > >> >> >> > have
> > > > >> >> >> > to
> > > > >> >> >> > implement a whole new on-demand VM support structure of =
some
> > > > sort
> > > > >> >> >> > for
> > > > >> >> >> > #2
> > > > >> >> >> > (which would be an extraordinary and impractical demand)?
> > > > >> >> >>
> > > > >> >> >> I did not mention VMs. Item #2 means a build, which runs
> > > > >> >> >> "test-patch"
> > > > >> >> >> target with the file specified by a user (instead of a jira
> > > > >> >> >> attachment).
> > > > >> >> >> When user clicks "Build Now" link a box is displayed where=
 the
> > > > user
> > > > >> >> >> can enter the file path containing the patch. This can be
> > > > specified
> > > > >> >> >> in
> > > > >> >> >> the Build Configuration under "This build is parameterized=
" by
> > > > >> >> >> choosing AddParameter / FileParameter. The build can run on
> > the
> > > > same
> > > > >> >> >> Windows machine as the nightly build.
> > > > >> >> >> Such build will let people test their patches for Windows =
on
> > > > Jenkins
> > > > >> >> >> if they don't posses a license for the right version of
> > Windows.
> > > > >> >> >> I hope this will not turn into extraordinary or impractical
> > > > effort.
> > > > >> >> >>
> > > > >> >> >> Thanks,
> > > > >> >> >> --Konst
> > > > >> >> >>
> > > > >> >> >> > Thanks,
> > > > >> >> >> > --Matt
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko
> > > > >> >> >> > <shv.hadoop@gmail.com>
> > > > >> >> >> > wrote:
> > > > >> >> >> >>
> > > > >> >> >> >> -1
> > > > >> >> >> >> We should have a CI infrastructure in place before we c=
an
> > > > commit
> > > > >> >> >> >> to
> > > > >> >> >> >> supporting Windows platform.
> > > > >> >> >> >>
> > > > >> >> >> >> Eric is right Win/Cygwin was supported since day one.
> > > > >> >> >> >> I had a Windows box under my desk running nightly builds
> > back
> > > > in
> > > > >> >> >> >> 2006-07.
> > > > >> >> >> >> People were irritated but I was filing windows bugs unt=
il
> > 0.22
> > > > >> >> >> >> release.
> > > > >> >> >> >> Times changing and I am glad to see wider support for W=
in
> > > > >> >> >> >> platform.
> > > > >> >> >> >>
> > > > >> >> >> >> But in order to make it work you guys need to put the CI
> > > > process
> > > > >> >> >> >> in
> > > > >> >> >> >> place
> > > > >> >> >> >>
> > > > >> >> >> >> 1. windows jenkins build: could be nightly or PreCommit.
> > > > >> >> >> >> - Nightly would mean that changes can be committed to t=
runk
> > > > based
> > > > >> >> >> >> on
> > > > >> >> >> >> linux PreCommit build. And people will file bugs if the
> > change
> > > > >> >> >> >> broke
> > > > >> >> >> >> Windows nightly build.
> > > > >> >> >> >> - PreCommit-win build will mean automatic reporting fai=
led
> > > > tests
> > > > >> >> >> >> to
> > > > >> >> >> >> respective jira blocking commits the same way as it is =
now
> > with
> > > > >> >> >> >> linux
> > > > >> >> >> >> PreCommit builds.
> > > > >> >> >> >> We should discuss which way is more efficient for
> > developers.
> > > > >> >> >> >>
> > > > >> >> >> >> 2. On-demand-windows Jenkins build.
> > > > >> >> >> >> I see it as a build to which I can attach my patch and =
the
> > > > build
> > > > >> >> >> >> will
> > > > >> >> >> >> run my changes on a dedicated windows box.
> > > > >> >> >> >> That way people can test their changes without having
> > personal
> > > > >> >> >> >> windows
> > > > >> >> >> >> nodes.
> > > > >> >> >> >>
> > > > >> >> >> >> I think this is the minimal set of requirement for us t=
o be
> > > > able
> > > > >> >> >> >> to
> > > > >> >> >> >> commit to the new platform.
> > > > >> >> >> >> Right now I see only one windows related build
> > > > >> >> >> >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/
> > > > >> >> >> >> Which was failing since Sept 8, 2012 and did not run in=
 the
> > > > last
> > > > >> >> >> >> month.
> > > > >> >> >> >>
> > > > >> >> >> >> Thanks,
> > > > >> >> >> >> --Konst
> > > > >> >> >> >>
> > > > >> >> >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler
> > > > >> >> >> >> <eric14@hortonworks.com> wrote:
> > > > >> >> >> >> > +1 (non-binding)
> > > > >> >> >> >> >
> > > > >> >> >> >> > A few of observations:
> > > > >> >> >> >> >
> > > > >> >> >> >> > - Windows has actually been a supported platform for
> > Hadoop
> > > > >> >> >> >> > since
> > > > >> >> >> >> > 0.1
> > > > >> >> >> >> > .
> > > > >> >> >> >> > Doug championed supporting windows then and we've
> > continued
> > > > to
> > > > >> >> >> >> > do
> > > > >> >> >> >> > it
> > > > >> >> >> >> > with
> > > > >> >> >> >> > varying vigor over time.  To my knowledge we've never
> > made a
> > > > >> >> >> >> > decision
> > > > >> >> >> >> > to
> > > > >> >> >> >> > drop windows support.  The change here is improving o=
ur
> > > > support
> > > > >> >> >> >> > and
> > > > >> >> >> >> > dropping
> > > > >> >> >> >> > the requirement of cigwin.  We had Nutch windows users
> > on the
> > > > >> >> >> >> > list
> > > > >> >> >> >> > in
> > > > >> >> >> >> > 2006
> > > > >> >> >> >> > and we've been supporting windows FS requirements sin=
ce
> > > > >> >> >> >> > inception.
> > > > >> >> >> >> >
> > > > >> >> >> >> > - A little pragmatism will go a long way.  As a commu=
nity
> > > > we've
> > > > >> >> >> >> > got
> > > > >> >> >> >> > to
> > > > >> >> >> >> > stay committed to keeping hadoop simple (so it does w=
ork
> > on
> > > > >> >> >> >> > many
> > > > >> >> >> >> > platforms)
> > > > >> >> >> >> > and extending it to take advantage of key emerging
> > > > OS/hardware
> > > > >> >> >> >> > features,
> > > > >> >> >> >> > such as containers, new FSs, virtualization, flash ...
> >  We
> > > > >> >> >> >> > should
> > > > >> >> >> >> > all
> > > > >> >> >> >> > plan
> > > > >> >> >> >> > to let new features & optimizations emerge that don't
> > work
> > > > >> >> >> >> > everywhere, if
> > > > >> >> >> >> > they are compelling and central to hadoop's mission of
> > being
> > > > >> >> >> >> > THE
> > > > >> >> >> >> > best
> > > > >> >> >> >> > fabric
> > > > >> >> >> >> > for storing and processing big data.
> > > > >> >> >> >> >
> > > > >> >> >> >> > - A UI project like KDE has to deal with the MANY
> > differences
> > > > >> >> >> >> > between
> > > > >> >> >> >> > windows and linux UI APIs.  Hadoop faces no such comp=
lex
> > > > >> >> >> >> > challenge
> > > > >> >> >> >> > and hence
> > > > >> >> >> >> > can be maintained from a single codeline IMO.  It is
> > mostly
> > > > >> >> >> >> > abstracted from
> > > > >> >> >> >> > the OS APIs via Java and our design choices.  Where it
> > is not
> > > > >> >> >> >> > we
> > > > >> >> >> >> > can
> > > > >> >> >> >> > continue to add plugable abstractions.
> > > > >> >> >> >> >
> > > > >> >> >> >
> > > > >> >> >> >
> > > > >> >> >
> > > > >> >> >
> > > > >> >
> > > > >> >
> > > > >
> > > > >
> > > >
> >
>=20
>=20
>=20
> --=20
> http://hortonworks.com/download/

--IrhDeMKUP4DT/M7F
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: Digital signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)

iF4EAREIAAYFAlFRBGwACgkQenyFlstYjhKMRwD/RhJCjnQ3hnrLQmPZMX+IdxdR
7dUx+pBt8hLC7ytyJtEA+wc4B4f1Eprhc21Sgg8gL8HB/5R6hp4ypIyAxRY/7sAt
=fQy8
-----END PGP SIGNATURE-----

--IrhDeMKUP4DT/M7F--