Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of suresh@hortonworks.com
 designates 209.85.128.50 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAPn_vTtvqYLUq+uBK+=sRFjkMKCUe2mpnH8MGBjBmbaHFFE5vQ@mail.gmail.com>
References: 
 <CADdVdVH7jhkYKV+1rBtHA-KD4zwTUnkKF9OVjwCJHkGZ8S9BUw@mail.gmail.com>
	<CD5384BD.6DDB%evans@yahoo-inc.com>
	<CAPn_vTtvqYLUq+uBK+=sRFjkMKCUe2mpnH8MGBjBmbaHFFE5vQ@mail.gmail.com>
Date: Wed, 27 Feb 2013 14:54:16 -0800
Message-ID: 
 <CADdVdVFY5N6x2xNHmBzyzLqXMWER_x5zHuFA-_iynOm-4SpqKQ@mail.gmail.com>
Subject: Re: [Vote] Merge branch-trunk-win to trunk
From: Suresh Srinivas <suresh@hortonworks.com>
To: "yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>
Cc: "hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>,
	"common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>,
	"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>
Content-Type: multipart/alternative; boundary=20cf3074b1285749a604d6bca757

--20cf3074b1285749a604d6bca757
Content-Type: text/plain; charset=ISO-8859-1

Thanks for raising good questions.

Currently the merge patch passes all the tests on Linux, hence
the proposal for merging the patch to trunk. But as Bobby, Harsh
and Eli pointed out, before declaring support for Windows, we need the
discussion on the following:

1. Precommit and development process
Jenkins infrastructure for Windows build will be made available.
Giri and Microsoft contributors have volunteered to help make
this happen.

With that we need to decide how our precommit process looks.
My inclination is to wait for +1 from precommit builds on
both the platforms to ensure no issues are introduced.
Thoughts?

2. Feature development impact
Some questions have been raised about would new features
need to be supported on both the platforms. Yes. I do not see a
reason why features cannot work on both the platforms, with
the exception of platform specific optimizations. This what Java
gives us.

3. Platform specific features/optimizations
As regards platform specific optimization, each platform can
evolve at its own pace and should not block progress of a
specific platform.

As indicated in my earlier email, there is a sizable number
of contributors to work on issues and support of Hadoop on Windows
platform. I am excited to see Hadoop reach the other large
part of server market.

Eli, as pointed out by you, the TODO items need to be addressed.
Also we realized we still need to add information on how to
build on Windows in BUILDING.txt. We will address this ASAP.
Giri and Matt have some expirience with this and should be able
to provide more information.


On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <eli@cloudera.com> wrote:

> Bobby raises some good questions.  A related one, since most current
> developers won't add Windows support for new features that are
> platform specific is it assumed that Windows development will either
> lag or will people actively work on keeping Windows up with the
> latest?  And vice versa in case Windows support is implemented first.
>
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).
>
> Thanks,
> Eli
>
> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <evans@yahoo-inc.com> wrote:
> > After this is merged in is Windows still going to be a second class
> > citizen but happens to work for more than just development or is it a
> > fully supported platform where if something breaks it can block a
> release?
> >  How do we as a community intend to keep Windows support from breaking?
> > We don't have any Jenkins slaves to be able to run nightly tests to
> > validate everything still compiles/runs.  This is not a blocker for me
> > because we often rely on individuals and groups to test Hadoop, but I do
> > think we need to have this discussion before we put it in.
> >
> > --Bobby
> >
> > On 2/26/13 4:55 PM, "Suresh Srinivas" <suresh@hortonworks.com> wrote:
> >
> >>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> >>I
> >>am happy to announce that we are ready for the merge.
> >>
> >>Here is a brief recap on the highlights of the work done:
> >>- Command-line scripts for the Hadoop surface area
> >>- Mapping the HDFS permissions model to Windows
> >>- Abstracted and reconciled mismatches around differences in Path
> >>semantics
> >>in Java and Windows
> >>- Native Task Controller for Windows
> >>- Implementation of a Block Placement Policy to support cloud
> >>environments,
> >>more specifically Azure.
> >>- Implementation of Hadoop native libraries for Windows (compression
> >>codecs, native I/O)
> >>- Several reliability issues, including race-conditions, intermittent
> test
> >>failures, resource leaks.
> >>- Several new unit test cases written for the above changes
> >>
> >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9
> >,
> >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> >>ported from branch-1-win to a branch based on trunk.
> >>
> >>For details of the testing done, please see the thread -
> >>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >>
> >>This was a large undertaking that involved developing code, testing the
> >>entire Hadoop stack, including scale tests. This is made possible only
> >>with
> >>the contribution from many many folks in the community. Following people
> >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur
> >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> Suresh
> >>Srinivas and Sanjay Radia. There are many others who contributed as well
> >>providing feedback and comments on numerous jiras.
> >>
> >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> >>
> >>Regards,
> >>Suresh
> >>
> >>
> >>
> >>
> >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >><mahadv@microsoft.com>wrote:
> >>
> >>> It is super exciting to look at the prospect of these changes being
> >>>merged
> >>> to trunk. Having Windows as one of the supported Hadoop platforms is a
> >>> fantastic opportunity both for the Hadoop project and Microsoft
> >>>customers.
> >>>
> >>> This work began around a year back when a few of us started with a
> basic
> >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> have
> >>> made significant progress in the following areas:
> >>> (PS: Some of these items are already included in Suresh's email, but
> >>> including again for completeness)
> >>>
> >>> - Command-line scripts for the Hadoop surface area
> >>> - Mapping the HDFS permissions model to Windows
> >>> - Abstracted and reconciled mismatches around differences in Path
> >>> semantics in Java and Windows
> >>> - Native Task Controller for Windows
> >>> - Implementation of a Block Placement Policy to support cloud
> >>> environments, more specifically Azure.
> >>> - Implementation of Hadoop native libraries for Windows (compression
> >>> codecs, native I/O) - Several reliability issues, including
> >>> race-conditions, intermittent test failures, resource leaks.
> >>> - Several new unit test cases written for the above changes
> >>>
> >>> In the process, we have closely engaged with the Apache open source
> >>> community and have got great support and assistance from the community
> >>>in
> >>> terms of contributing fixes, code review comments and commits.
> >>>
> >>> In addition, the Hadoop team at Microsoft has also made good progress
> in
> >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
> >>>of
> >>> these changes have already been committed to the respective trunks with
> >>> help from various committers and contributors. It is great to see the
> >>> commitment of the community to support multiple platforms, and we look
> >>> forward to the day when a developer/customer is able to successfully
> >>>deploy
> >>> a complete solution stack based on Apache Hadoop releases.
> >>>
> >>> Next Steps:
> >>>
> >>> All of the above changes are part of the Windows Azure HDInsight and
> >>> HDInsight Server products from Microsoft. We have successfully
> >>>on-boarded
> >>> several internal customers and have been running production workloads
> on
> >>> Windows Azure HDInsight. Our vision is to create a big data platform
> >>>based
> >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> >>> solution that anyone can use to solve their biggest data challenges.
> >>>
> >>> As an immediate next step, we would like to have a discussion around
> how
> >>> we can ensure that the quality of the mainline Hadoop branches on
> >>>Windows
> >>> is maintained. To this end, we would like to get to the state where we
> >>>have
> >>> pre-checkin validation gates and nightly test runs enabled on Windows.
> >>>If
> >>> you have any suggestions around this, please do send an email.  We are
> >>> committed to helping sustain the long-term quality of Hadoop on both
> >>>Linux
> >>> and Windows.
> >>>
> >>> We sincerely thank the community for their contribution and support so
> >>> far. And hope to continue having a close engagement in the future.
> >>>
> >>> -Microsoft HDInsight Team
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >>> Sent: Thursday, February 7, 2013 5:42 PM
> >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >>> Subject: Heads up - merge branch-trunk-win to trunk
> >>>
> >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> >>>The
> >>> goal was to make Hadoop natively integrated, full-featured, and
> >>>performance
> >>> and scalability tuned on Windows Server or Windows Azure.
> >>> We are happy to announce that a lot of progress has been made in this
> >>> regard.
> >>>
> >>> Initial work started in a feature branch, branch-1-win, based on
> >>>branch-1.
> >>> The details related to the work done in the branch can be seen in
> >>> CHANGES.txt<
> >>>
> >>>
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
> >>>branch-1-win.txt?view=markup
> >>> >.
> >>> This work has been ported to a branch, branch-trunk-win, based on
> trunk.
> >>> Merge patch for this is available on
> >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >>> .
> >>>
> >>> Highlights of the work done so far:
> >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes
> >>> handle differences in platforms related to path names, process/task
> >>> management etc.
> >>> 2. Addition of winutils tools for managing file permissions and
> >>>ownership,
> >>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> >>>and
> >>> process/task management.
> >>> 3. Added cmd scripts equivalent to existing shell scripts
> >>> hadoop-daemon.sh, start and stop scripts.
> >>> 4. Addition of block placement policy implemnation to support cloud
> >>> enviroment, more specifically Azure.
> >>>
> >>> We are very close to wrapping up the work in branch-trunk-win and
> >>>getting
> >>> ready for a merge. Currently the merge patch is passing close to 100%
> of
> >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> >>>into
> >>> trunk.
> >>>
> >>> Next steps:
> >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> >>> completes and precommit build is clean.
> >>> 2. Start a discussion on adding Jenkins precommit builds on windows and
> >>> how to integrate that with the existing commit process.
> >>>
> >>> Let me know if you have any questions.
> >>>
> >>> Regards,
> >>> Suresh
> >>>
> >>>
> >>
> >>
> >>--
> >>http://hortonworks.com/download/
> >
>


-- 
http://hortonworks.com/download/

--20cf3074b1285749a604d6bca757--