hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chris Nauroth <cnaur...@hortonworks.com>
Subject Re: [Vote] Merge branch-trunk-win to trunk
Date Wed, 27 Feb 2013 06:05:42 GMT
+1 (non-binding)

I've been testing and patching this branch for the past several months, and
I believe we've reached stability for a merge to trunk.  I want to point
out once again that the branch has been tested on Linux to build confidence
that regressions were not introduced on existing platforms.  I also want to
second the comments from Bikas about how wonderful the collaboration has
been!

Thank you,
--Chris


On Tue, Feb 26, 2013 at 4:30 PM, Bikas Saha <bikas@hortonworks.com> wrote:

> +1
>
> As someone who has been part of this effort from inception, I am glad that
> we have reached this stable state in the project on both branches of
> Hadoop.
> It has been a great collaboration across teams and engineers and opens up
> Hadoop to a whole new set of deployments and developers!
>
> Bikas
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Tuesday, February 26, 2013 2:56 PM
> To: common-dev@hadoop.apache.org
> Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
> mapreduce-dev@hadoop.apache.org
> Subject: [Vote] Merge branch-trunk-win to trunk
>
> I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> I am happy to announce that we are ready for the merge.
>
> Here is a brief recap on the highlights of the work done:
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O)
> - Several reliability issues, including race-conditions, intermittent test
> failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> Please find the details of the work in CHANGES.branch-trunk-win.txt -
> Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
> and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> ported from branch-1-win to a branch based on trunk.
>
> For details of the testing done, please see the thread -
> http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> https://issues.apache.org/jira/browse/HADOOP-8562>.
>
> This was a large undertaking that involved developing code, testing the
> entire Hadoop stack, including scale tests. This is made possible only
> with the contribution from many many folks in the community. Following
> people contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
> Saha, Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
> Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
> Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
> Nicholas Sze, Suresh Srinivas and Sanjay Radia. There are many others who
> contributed as well providing feedback and comments on numerous jiras.
>
> The vote will run for seven days and will end on March 5, 6:00PM PST.
>
> Regards,
> Suresh
>
>
>
>
> On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> <mahadv@microsoft.com>wrote:
>
> > It is super exciting to look at the prospect of these changes being
> > merged to trunk. Having Windows as one of the supported Hadoop
> > platforms is a fantastic opportunity both for the Hadoop project and
> Microsoft customers.
> >
> > This work began around a year back when a few of us started with a
> > basic port of Hadoop on Windows. Ever since, the Hadoop team in
> > Microsoft have made significant progress in the following areas:
> > (PS: Some of these items are already included in Suresh's email, but
> > including again for completeness)
> >
> > - Command-line scripts for the Hadoop surface area
> > - Mapping the HDFS permissions model to Windows
> > - Abstracted and reconciled mismatches around differences in Path
> > semantics in Java and Windows
> > - Native Task Controller for Windows
> > - Implementation of a Block Placement Policy to support cloud
> > environments, more specifically Azure.
> > - Implementation of Hadoop native libraries for Windows (compression
> > codecs, native I/O) - Several reliability issues, including
> > race-conditions, intermittent test failures, resource leaks.
> > - Several new unit test cases written for the above changes
> >
> > In the process, we have closely engaged with the Apache open source
> > community and have got great support and assistance from the community
> > in terms of contributing fixes, code review comments and commits.
> >
> > In addition, the Hadoop team at Microsoft has also made good progress
> > in other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> > Many of these changes have already been committed to the respective
> > trunks with help from various committers and contributors. It is great
> > to see the commitment of the community to support multiple platforms,
> > and we look forward to the day when a developer/customer is able to
> > successfully deploy a complete solution stack based on Apache Hadoop
> releases.
> >
> > Next Steps:
> >
> > All of the above changes are part of the Windows Azure HDInsight and
> > HDInsight Server products from Microsoft. We have successfully
> > on-boarded several internal customers and have been running production
> > workloads on Windows Azure HDInsight. Our vision is to create a big
> > data platform based on Hadoop, and we are committed to helping make
> > Hadoop a world-class solution that anyone can use to solve their biggest
> data challenges.
> >
> > As an immediate next step, we would like to have a discussion around
> > how we can ensure that the quality of the mainline Hadoop branches on
> > Windows is maintained. To this end, we would like to get to the state
> > where we have pre-checkin validation gates and nightly test runs
> > enabled on Windows. If you have any suggestions around this, please do
> > send an email.  We are committed to helping sustain the long-term
> > quality of Hadoop on both Linux and Windows.
> >
> > We sincerely thank the community for their contribution and support so
> > far. And hope to continue having a close engagement in the future.
> >
> > -Microsoft HDInsight Team
> >
> >
> > -----Original Message-----
> > From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > Sent: Thursday, February 7, 2013 5:42 PM
> > To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > Subject: Heads up - merge branch-trunk-win to trunk
> >
> > The support for Hadoop on Windows was proposed in HADOOP-8079<
> > https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> > The goal was to make Hadoop natively integrated, full-featured, and
> > performance and scalability tuned on Windows Server or Windows Azure.
> > We are happy to announce that a lot of progress has been made in this
> > regard.
> >
> > Initial work started in a feature branch, branch-1-win, based on
> branch-1.
> > The details related to the work done in the branch can be seen in
> > CHANGES.txt<
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANG
> > ES.branch-1-win.txt?view=markup
> > >.
> > This work has been ported to a branch, branch-trunk-win, based on trunk.
> > Merge patch for this is available on
> > HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > .
> >
> > Highlights of the work done so far:
> > 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes handle differences in platforms related to path names,
> > process/task management etc.
> > 2. Addition of winutils tools for managing file permissions and
> > ownership, user group mapping, hardlinks, symbolic links, chmod, disk
> > utilization, and process/task management.
> > 3. Added cmd scripts equivalent to existing shell scripts
> > hadoop-daemon.sh, start and stop scripts.
> > 4. Addition of block placement policy implemnation to support cloud
> > enviroment, more specifically Azure.
> >
> > We are very close to wrapping up the work in branch-trunk-win and
> > getting ready for a merge. Currently the merge patch is passing close
> > to 100% of unit tests on Linux. Soon I will call for a vote to merge
> > this branch into trunk.
> >
> > Next steps:
> > 1. Call for vote to merge branch-trunk-win to trunk, when the work
> > completes and precommit build is clean.
> > 2. Start a discussion on adding Jenkins precommit builds on windows
> > and how to integrate that with the existing commit process.
> >
> > Let me know if you have any questions.
> >
> > Regards,
> > Suresh
> >
> >
>
>
> --
> http://hortonworks.com/download/
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message