Mailing-List: contact hdfs-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of kannakar@microsoft.com
 designates 207.46.100.30 as permitted sender)
From: Kanna Karanam <kannakar@microsoft.com>
To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
CC: "yarn-dev@hadoop.apache.org" <yarn-dev@hadoop.apache.org>,
	"hdfs-dev@hadoop.apache.org" <hdfs-dev@hadoop.apache.org>,
	"mapreduce-dev@hadoop.apache.org" <mapreduce-dev@hadoop.apache.org>
Subject: RE: [Vote] Merge branch-trunk-win to trunk
Thread-Topic: [Vote] Merge branch-trunk-win to trunk
Thread-Index: AQHOFeg/wcbZJLoT00a2IG6+4IY5+5iPpQJA
Date: Thu, 28 Feb 2013 19:26:38 +0000
Message-ID: 
 <9989D204521BF64FA72B12A739FAD15B27F9FF73@TK5EX14MBXC252.redmond.corp.microsoft.com>
References: 
 <CADdVdVH7jhkYKV+1rBtHA-KD4zwTUnkKF9OVjwCJHkGZ8S9BUw@mail.gmail.com>
 <CAOcaW6DAv1bj+_7_u=nzhpC5eW7aHuLzb8zQftajdOHSOKeRtw@mail.gmail.com>
In-Reply-To: 
 <CAOcaW6DAv1bj+_7_u=nzhpC5eW7aHuLzb8zQftajdOHSOKeRtw@mail.gmail.com>
Accept-Language: en-US
Content-Language: en-US
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0

+1 non-binding

I am playing with it for several months in a multi-node Windows cluster env=
ironment and found it is very stable. I am sure that it can help us to brin=
g more developers like me (JAVA & Windows Developers) to contribute more an=
d help the Hadoop customer & developer communities.

Thanks,
Kanna

-----Original Message-----
From: Raja Aluri [mailto:raja@cmbasics.com]=20
Sent: Thursday, February 28, 2013 11:17 AM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; mapreduce-dev@h=
adoop.apache.org
Subject: Re: [Vote] Merge branch-trunk-win to trunk

+1 non-binding
Nice to see that this work is going to trunk.

Raja  Aluri


On Tue, Feb 26, 2013 at 2:55 PM, Suresh Srinivas <suresh@hortonworks.com>wr=
ote:

> I had posted heads up about merging branch-trunk-win to trunk on Feb=20
> 8th. I am happy to announce that we are ready for the merge.
>
> Here is a brief recap on the highlights of the work done:
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path=20
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud=20
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression=20
> codecs, native I/O)
> - Several reliability issues, including race-conditions, intermittent=20
> test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> Please find the details of the work in CHANGES.branch-trunk-win.txt -=20
> Common changes<http://bit.ly/Xe7Ynv>, HDFS=20
> changes<http://bit.ly/13QOSo9>, and YARN and MapReduce changes=20
> <http://bit.ly/128zzMt>. This is the work ported from branch-1-win to a b=
ranch based on trunk.
>
> For details of the testing done, please see the thread -=20
> http://bit.ly/WpavJ4. Merge patch for this is available on=20
> HADOOP-8562< https://issues.apache.org/jira/browse/HADOOP-8562>.
>
> This was a large undertaking that involved developing code, testing=20
> the entire Hadoop stack, including scale tests. This is made possible=20
> only with the contribution from many many folks in the community.=20
> Following people contributed to this work: Ivan Mitic, Chuan Liu,=20
> Ramya Sunil, Bikas Saha, Kanna Karanam, John Gordon, Brandon Li, Chris=20
> Nauroth, David Lao, Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz,=20
> Mike Liddell, Jing Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja=20
> Aluri, Giridharan Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp,=20
> Arun Murthy, Tsz-Wo Nicholas Sze, Suresh Srinivas and Sanjay Radia.=20
> There are many others who contributed as well providing feedback and comm=
ents on numerous jiras.
>
> The vote will run for seven days and will end on March 5, 6:00PM PST.
>
> Regards,
> Suresh
>
>
>
>
> On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> <mahadv@microsoft.com>wrote:
>
> > It is super exciting to look at the prospect of these changes being
> merged
> > to trunk. Having Windows as one of the supported Hadoop platforms is=20
> > a fantastic opportunity both for the Hadoop project and Microsoft
> customers.
> >
> > This work began around a year back when a few of us started with a=20
> > basic port of Hadoop on Windows. Ever since, the Hadoop team in=20
> > Microsoft have made significant progress in the following areas:
> > (PS: Some of these items are already included in Suresh's email, but=20
> > including again for completeness)
> >
> > - Command-line scripts for the Hadoop surface area
> > - Mapping the HDFS permissions model to Windows
> > - Abstracted and reconciled mismatches around differences in Path=20
> > semantics in Java and Windows
> > - Native Task Controller for Windows
> > - Implementation of a Block Placement Policy to support cloud=20
> > environments, more specifically Azure.
> > - Implementation of Hadoop native libraries for Windows (compression=20
> > codecs, native I/O) - Several reliability issues, including=20
> > race-conditions, intermittent test failures, resource leaks.
> > - Several new unit test cases written for the above changes
> >
> > In the process, we have closely engaged with the Apache open source=20
> > community and have got great support and assistance from the=20
> > community in terms of contributing fixes, code review comments and comm=
its.
> >
> > In addition, the Hadoop team at Microsoft has also made good=20
> > progress in other projects including Hive, Pig, Sqoop, Oozie, HCat=20
> > and HBase. Many of these changes have already been committed to the=20
> > respective trunks with help from various committers and=20
> > contributors. It is great to see the commitment of the community to=20
> > support multiple platforms, and we look forward to the day when a=20
> > developer/customer is able to successfully
> deploy
> > a complete solution stack based on Apache Hadoop releases.
> >
> > Next Steps:
> >
> > All of the above changes are part of the Windows Azure HDInsight and=20
> > HDInsight Server products from Microsoft. We have successfully=20
> > on-boarded several internal customers and have been running=20
> > production workloads on Windows Azure HDInsight. Our vision is to=20
> > create a big data platform
> based
> > on Hadoop, and we are committed to helping make Hadoop a world-class=20
> > solution that anyone can use to solve their biggest data challenges.
> >
> > As an immediate next step, we would like to have a discussion around=20
> > how we can ensure that the quality of the mainline Hadoop branches=20
> > on Windows is maintained. To this end, we would like to get to the=20
> > state where we
> have
> > pre-checkin validation gates and nightly test runs enabled on=20
> > Windows. If you have any suggestions around this, please do send an=20
> > email.  We are committed to helping sustain the long-term quality of=20
> > Hadoop on both
> Linux
> > and Windows.
> >
> > We sincerely thank the community for their contribution and support=20
> > so far. And hope to continue having a close engagement in the future.
> >
> > -Microsoft HDInsight Team
> >
> >
> > -----Original Message-----
> > From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > Sent: Thursday, February 7, 2013 5:42 PM
> > To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;=20
> > hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Heads up - merge branch-trunk-win to trunk
> >
> > The support for Hadoop on Windows was proposed in HADOOP-8079<=20
> > https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> The
> > goal was to make Hadoop natively integrated, full-featured, and
> performance
> > and scalability tuned on Windows Server or Windows Azure.
> > We are happy to announce that a lot of progress has been made in=20
> > this regard.
> >
> > Initial work started in a feature branch, branch-1-win, based on
> branch-1.
> > The details related to the work done in the branch can be seen in=20
> > CHANGES.txt<
> >
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANG
> ES.branch-1-win.txt?view=3Dmarkup
> > >.
> > This work has been ported to a branch, branch-trunk-win, based on trunk=
.
> > Merge patch for this is available on=20
> > HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > .
> >
> > Highlights of the work done so far:
> > 1. Necessary changes in Hadoop to run natively on Windows. These=20
> > changes handle differences in platforms related to path names,=20
> > process/task management etc.
> > 2. Addition of winutils tools for managing file permissions and
> ownership,
> > user group mapping, hardlinks, symbolic links, chmod, disk=20
> > utilization,
> and
> > process/task management.
> > 3. Added cmd scripts equivalent to existing shell scripts=20
> > hadoop-daemon.sh, start and stop scripts.
> > 4. Addition of block placement policy implemnation to support cloud=20
> > enviroment, more specifically Azure.
> >
> > We are very close to wrapping up the work in branch-trunk-win and=20
> > getting ready for a merge. Currently the merge patch is passing=20
> > close to 100% of unit tests on Linux. Soon I will call for a vote to=20
> > merge this branch
> into
> > trunk.
> >
> > Next steps:
> > 1. Call for vote to merge branch-trunk-win to trunk, when the work=20
> > completes and precommit build is clean.
> > 2. Start a discussion on adding Jenkins precommit builds on windows=20
> > and how to integrate that with the existing commit process.
> >
> > Let me know if you have any questions.
> >
> > Regards,
> > Suresh
> >
> >
>
>
> --
> http://hortonworks.com/download/
>