hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Evans <ev...@yahoo-inc.com>
Subject Re: [Vote] Merge branch-trunk-win to trunk
Date Wed, 27 Feb 2013 16:17:56 GMT
After this is merged in is Windows still going to be a second class
citizen but happens to work for more than just development or is it a
fully supported platform where if something breaks it can block a release?
 How do we as a community intend to keep Windows support from breaking?
We don't have any Jenkins slaves to be able to run nightly tests to
validate everything still compiles/runs.  This is not a blocker for me
because we often rely on individuals and groups to test Hadoop, but I do
think we need to have this discussion before we put it in.

--Bobby

On 2/26/13 4:55 PM, "Suresh Srinivas" <suresh@hortonworks.com> wrote:

>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
>I
>am happy to announce that we are ready for the merge.
>
>Here is a brief recap on the highlights of the work done:
>- Command-line scripts for the Hadoop surface area
>- Mapping the HDFS permissions model to Windows
>- Abstracted and reconciled mismatches around differences in Path
>semantics
>in Java and Windows
>- Native Task Controller for Windows
>- Implementation of a Block Placement Policy to support cloud
>environments,
>more specifically Azure.
>- Implementation of Hadoop native libraries for Windows (compression
>codecs, native I/O)
>- Several reliability issues, including race-conditions, intermittent test
>failures, resource leaks.
>- Several new unit test cases written for the above changes
>
>Please find the details of the work in CHANGES.branch-trunk-win.txt -
>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
>ported from branch-1-win to a branch based on trunk.
>
>For details of the testing done, please see the thread -
>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
>https://issues.apache.org/jira/browse/HADOOP-8562>.
>
>This was a large undertaking that involved developing code, testing the
>entire Hadoop stack, including scale tests. This is made possible only
>with
>the contribution from many many folks in the community. Following people
>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao, Sumadhur
>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze, Suresh
>Srinivas and Sanjay Radia. There are many others who contributed as well
>providing feedback and comments on numerous jiras.
>
>The vote will run for seven days and will end on March 5, 6:00PM PST.
>
>Regards,
>Suresh
>
>
>
>
>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
><mahadv@microsoft.com>wrote:
>
>> It is super exciting to look at the prospect of these changes being
>>merged
>> to trunk. Having Windows as one of the supported Hadoop platforms is a
>> fantastic opportunity both for the Hadoop project and Microsoft
>>customers.
>>
>> This work began around a year back when a few of us started with a basic
>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft have
>> made significant progress in the following areas:
>> (PS: Some of these items are already included in Suresh's email, but
>> including again for completeness)
>>
>> - Command-line scripts for the Hadoop surface area
>> - Mapping the HDFS permissions model to Windows
>> - Abstracted and reconciled mismatches around differences in Path
>> semantics in Java and Windows
>> - Native Task Controller for Windows
>> - Implementation of a Block Placement Policy to support cloud
>> environments, more specifically Azure.
>> - Implementation of Hadoop native libraries for Windows (compression
>> codecs, native I/O) - Several reliability issues, including
>> race-conditions, intermittent test failures, resource leaks.
>> - Several new unit test cases written for the above changes
>>
>> In the process, we have closely engaged with the Apache open source
>> community and have got great support and assistance from the community
>>in
>> terms of contributing fixes, code review comments and commits.
>>
>> In addition, the Hadoop team at Microsoft has also made good progress in
>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
>>of
>> these changes have already been committed to the respective trunks with
>> help from various committers and contributors. It is great to see the
>> commitment of the community to support multiple platforms, and we look
>> forward to the day when a developer/customer is able to successfully
>>deploy
>> a complete solution stack based on Apache Hadoop releases.
>>
>> Next Steps:
>>
>> All of the above changes are part of the Windows Azure HDInsight and
>> HDInsight Server products from Microsoft. We have successfully
>>on-boarded
>> several internal customers and have been running production workloads on
>> Windows Azure HDInsight. Our vision is to create a big data platform
>>based
>> on Hadoop, and we are committed to helping make Hadoop a world-class
>> solution that anyone can use to solve their biggest data challenges.
>>
>> As an immediate next step, we would like to have a discussion around how
>> we can ensure that the quality of the mainline Hadoop branches on
>>Windows
>> is maintained. To this end, we would like to get to the state where we
>>have
>> pre-checkin validation gates and nightly test runs enabled on Windows.
>>If
>> you have any suggestions around this, please do send an email.  We are
>> committed to helping sustain the long-term quality of Hadoop on both
>>Linux
>> and Windows.
>>
>> We sincerely thank the community for their contribution and support so
>> far. And hope to continue having a close engagement in the future.
>>
>> -Microsoft HDInsight Team
>>
>>
>> -----Original Message-----
>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
>> Sent: Thursday, February 7, 2013 5:42 PM
>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
>> Subject: Heads up - merge branch-trunk-win to trunk
>>
>> The support for Hadoop on Windows was proposed in HADOOP-8079<
>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
>>The
>> goal was to make Hadoop natively integrated, full-featured, and
>>performance
>> and scalability tuned on Windows Server or Windows Azure.
>> We are happy to announce that a lot of progress has been made in this
>> regard.
>>
>> Initial work started in a feature branch, branch-1-win, based on
>>branch-1.
>> The details related to the work done in the branch can be seen in
>> CHANGES.txt<
>> 
>>http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
>>branch-1-win.txt?view=markup
>> >.
>> This work has been ported to a branch, branch-trunk-win, based on trunk.
>> Merge patch for this is available on
>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
>> .
>>
>> Highlights of the work done so far:
>> 1. Necessary changes in Hadoop to run natively on Windows. These changes
>> handle differences in platforms related to path names, process/task
>> management etc.
>> 2. Addition of winutils tools for managing file permissions and
>>ownership,
>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
>>and
>> process/task management.
>> 3. Added cmd scripts equivalent to existing shell scripts
>> hadoop-daemon.sh, start and stop scripts.
>> 4. Addition of block placement policy implemnation to support cloud
>> enviroment, more specifically Azure.
>>
>> We are very close to wrapping up the work in branch-trunk-win and
>>getting
>> ready for a merge. Currently the merge patch is passing close to 100% of
>> unit tests on Linux. Soon I will call for a vote to merge this branch
>>into
>> trunk.
>>
>> Next steps:
>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
>> completes and precommit build is clean.
>> 2. Start a discussion on adding Jenkins precommit builds on windows and
>> how to integrate that with the existing commit process.
>>
>> Let me know if you have any questions.
>>
>> Regards,
>> Suresh
>>
>>
>
>
>-- 
>http://hortonworks.com/download/


Mime
View raw message