Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of mahadv@microsoft.com
 designates 207.46.100.30 as permitted sender)
Received-SPF: softfail (mail165-tx2: transitioning domain of microsoft.com
 does not designate 157.56.240.21 as permitted sender)
 client-ip=157.56.240.21; envelope-from=mahadv@microsoft.com;
 helo=BL2PRD0310HT004.namprd03.prod.outlook.com ;.outlook.com ;
From: Mahadevan Venkatraman <mahadv@microsoft.com>
To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>,
	"yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>,
	"hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>,
	"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>
Subject: RE: Heads up - merge branch-trunk-win to trunk
Thread-Topic: Heads up - merge branch-trunk-win to trunk
Thread-Index: AQHOBZ2xonAQdumCH0K6Q7DFAFBONZhvOF8g
Date: Fri, 8 Feb 2013 02:41:47 +0000
Message-ID: 
 <d04aea7b279849eba5888c0fc2a5e8a7@BLUPR03MB018.namprd03.prod.outlook.com>
References: 
 <CADdVdVE6nTUqmKcHMSdkFrHWYqhrVb6Kbp30JY6NLPyBjFWoHQ@mail.gmail.com>
In-Reply-To: 
 <CADdVdVE6nTUqmKcHMSdkFrHWYqhrVb6Kbp30JY6NLPyBjFWoHQ@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

It is super exciting to look at the prospect of these changes being merged =
to trunk. Having Windows as one of the supported Hadoop platforms is a fant=
astic opportunity both for the Hadoop project and Microsoft customers.

This work began around a year back when a few of us started with a basic po=
rt of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have made=
 significant progress in the following areas:
(PS: Some of these items are already included in Suresh's email, but includ=
ing again for completeness)

- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path semantics=
 in Java and Windows
- Native Task Controller for Windows=20
- Implementation of a Block Placement Policy to support cloud environments,=
 more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression codecs=
, native I/O) - Several reliability issues, including race-conditions, inte=
rmittent test failures, resource leaks.
- Several new unit test cases written for the above changes

In the process, we have closely engaged with the Apache open source communi=
ty and have got great support and assistance from the community in terms of=
 contributing fixes, code review comments and commits.=20

In addition, the Hadoop team at Microsoft has also made good progress in ot=
her projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many of the=
se changes have already been committed to the respective trunks with help f=
rom various committers and contributors. It is great to see the commitment =
of the community to support multiple platforms, and we look forward to the =
day when a developer/customer is able to successfully deploy a complete sol=
ution stack based on Apache Hadoop releases.

Next Steps:

All of the above changes are part of the Windows Azure HDInsight and HDInsi=
ght Server products from Microsoft. We have successfully on-boarded several=
 internal customers and have been running production workloads on Windows A=
zure HDInsight. Our vision is to create a big data platform based on Hadoop=
, and we are committed to helping make Hadoop a world-class solution that a=
nyone can use to solve their biggest data challenges.=20

As an immediate next step, we would like to have a discussion around how we=
 can ensure that the quality of the mainline Hadoop branches on Windows is =
maintained. To this end, we would like to get to the state where we have pr=
e-checkin validation gates and nightly test runs enabled on Windows. If you=
 have any suggestions around this, please do send an email.  We are committ=
ed to helping sustain the long-term quality of Hadoop on both Linux and Win=
dows.

We sincerely thank the community for their contribution and support so far.=
 And hope to continue having a close engagement in the future.

-Microsoft HDInsight Team


-----Original Message-----
From: Suresh Srinivas [mailto:suresh@hortonworks.com]=20
Sent: Thursday, February 7, 2013 5:42 PM
To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org; hdfs-dev@hado=
op.apache.org; mapreduce-dev@hadoop.apache.org
Subject: Heads up - merge branch-trunk-win to trunk

The support for Hadoop on Windows was proposed in HADOOP-8079<https://issue=
s.apache.org/jira/browse/HADOOP-8079> almost a year ago. The goal was to ma=
ke Hadoop natively integrated, full-featured, and performance and scalabili=
ty tuned on Windows Server or Windows Azure.
We are happy to announce that a lot of progress has been made in this regar=
d.

Initial work started in a feature branch, branch-1-win, based on branch-1.
The details related to the work done in the branch can be seen in CHANGES.t=
xt<http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES=
.branch-1-win.txt?view=3Dmarkup>.
This work has been ported to a branch, branch-trunk-win, based on trunk.
Merge patch for this is available on
HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
.

Highlights of the work done so far:
1. Necessary changes in Hadoop to run natively on Windows. These changes ha=
ndle differences in platforms related to path names, process/task managemen=
t etc.
2. Addition of winutils tools for managing file permissions and ownership, =
user group mapping, hardlinks, symbolic links, chmod, disk utilization, and=
 process/task management.
3. Added cmd scripts equivalent to existing shell scripts hadoop-daemon.sh,=
 start and stop scripts.
4. Addition of block placement policy implemnation to support cloud envirom=
ent, more specifically Azure.

We are very close to wrapping up the work in branch-trunk-win and getting r=
eady for a merge. Currently the merge patch is passing close to 100% of uni=
t tests on Linux. Soon I will call for a vote to merge this branch into tru=
nk.

Next steps:
1. Call for vote to merge branch-trunk-win to trunk, when the work complete=
s and precommit build is clean.
2. Start a discussion on adding Jenkins precommit builds on windows and how=
 to integrate that with the existing commit process.

Let me know if you have any questions.

Regards,
Suresh