hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arpit Agarwal <aagar...@hortonworks.com>
Subject Re: [Vote] Merge branch-trunk-win to trunk
Date Wed, 27 Feb 2013 22:08:22 GMT
+1 non-binding.

I have extensively tested this on both Windows and Linux over the last few
months.

Thanks,
-Arpit

On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <eli@cloudera.com> wrote:

> Bobby raises some good questions.  A related one, since most current
> developers won't add Windows support for new features that are
> platform specific is it assumed that Windows development will either
> lag or will people actively work on keeping Windows up with the
> latest?  And vice versa in case Windows support is implemented first.
>
> Is there a jira for resolving the outstanding TODOs in the code base
> (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> which is great (just did a quick diff and grep).
>
> Thanks,
> Eli
>
> On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <evans@yahoo-inc.com> wrote:
> > After this is merged in is Windows still going to be a second class
> > citizen but happens to work for more than just development or is it a
> > fully supported platform where if something breaks it can block a
> release?
> >  How do we as a community intend to keep Windows support from breaking?
> > We don't have any Jenkins slaves to be able to run nightly tests to
> > validate everything still compiles/runs.  This is not a blocker for me
> > because we often rely on individuals and groups to test Hadoop, but I do
> > think we need to have this discussion before we put it in.
> >
> > --Bobby
> >
> > On 2/26/13 4:55 PM, "Suresh Srinivas" <suresh@hortonworks.com> wrote:
> >
> >>I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
> >>I
> >>am happy to announce that we are ready for the merge.
> >>
> >>Here is a brief recap on the highlights of the work done:
> >>- Command-line scripts for the Hadoop surface area
> >>- Mapping the HDFS permissions model to Windows
> >>- Abstracted and reconciled mismatches around differences in Path
> >>semantics
> >>in Java and Windows
> >>- Native Task Controller for Windows
> >>- Implementation of a Block Placement Policy to support cloud
> >>environments,
> >>more specifically Azure.
> >>- Implementation of Hadoop native libraries for Windows (compression
> >>codecs, native I/O)
> >>- Several reliability issues, including race-conditions, intermittent
> test
> >>failures, resource leaks.
> >>- Several new unit test cases written for the above changes
> >>
> >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9
> >,
> >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
> >>ported from branch-1-win to a branch based on trunk.
> >>
> >>For details of the testing done, please see the thread -
> >>http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
> >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> >>
> >>This was a large undertaking that involved developing code, testing the
> >>entire Hadoop stack, including scale tests. This is made possible only
> >>with
> >>the contribution from many many folks in the community. Following people
> >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas Saha,
> >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> Sumadhur
> >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao, Thejas
> >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> Suresh
> >>Srinivas and Sanjay Radia. There are many others who contributed as well
> >>providing feedback and comments on numerous jiras.
> >>
> >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> >>
> >>Regards,
> >>Suresh
> >>
> >>
> >>
> >>
> >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> >><mahadv@microsoft.com>wrote:
> >>
> >>> It is super exciting to look at the prospect of these changes being
> >>>merged
> >>> to trunk. Having Windows as one of the supported Hadoop platforms is a
> >>> fantastic opportunity both for the Hadoop project and Microsoft
> >>>customers.
> >>>
> >>> This work began around a year back when a few of us started with a
> basic
> >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> have
> >>> made significant progress in the following areas:
> >>> (PS: Some of these items are already included in Suresh's email, but
> >>> including again for completeness)
> >>>
> >>> - Command-line scripts for the Hadoop surface area
> >>> - Mapping the HDFS permissions model to Windows
> >>> - Abstracted and reconciled mismatches around differences in Path
> >>> semantics in Java and Windows
> >>> - Native Task Controller for Windows
> >>> - Implementation of a Block Placement Policy to support cloud
> >>> environments, more specifically Azure.
> >>> - Implementation of Hadoop native libraries for Windows (compression
> >>> codecs, native I/O) - Several reliability issues, including
> >>> race-conditions, intermittent test failures, resource leaks.
> >>> - Several new unit test cases written for the above changes
> >>>
> >>> In the process, we have closely engaged with the Apache open source
> >>> community and have got great support and assistance from the community
> >>>in
> >>> terms of contributing fixes, code review comments and commits.
> >>>
> >>> In addition, the Hadoop team at Microsoft has also made good progress
> in
> >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase. Many
> >>>of
> >>> these changes have already been committed to the respective trunks with
> >>> help from various committers and contributors. It is great to see the
> >>> commitment of the community to support multiple platforms, and we look
> >>> forward to the day when a developer/customer is able to successfully
> >>>deploy
> >>> a complete solution stack based on Apache Hadoop releases.
> >>>
> >>> Next Steps:
> >>>
> >>> All of the above changes are part of the Windows Azure HDInsight and
> >>> HDInsight Server products from Microsoft. We have successfully
> >>>on-boarded
> >>> several internal customers and have been running production workloads
> on
> >>> Windows Azure HDInsight. Our vision is to create a big data platform
> >>>based
> >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> >>> solution that anyone can use to solve their biggest data challenges.
> >>>
> >>> As an immediate next step, we would like to have a discussion around
> how
> >>> we can ensure that the quality of the mainline Hadoop branches on
> >>>Windows
> >>> is maintained. To this end, we would like to get to the state where we
> >>>have
> >>> pre-checkin validation gates and nightly test runs enabled on Windows.
> >>>If
> >>> you have any suggestions around this, please do send an email.  We are
> >>> committed to helping sustain the long-term quality of Hadoop on both
> >>>Linux
> >>> and Windows.
> >>>
> >>> We sincerely thank the community for their contribution and support so
> >>> far. And hope to continue having a close engagement in the future.
> >>>
> >>> -Microsoft HDInsight Team
> >>>
> >>>
> >>> -----Original Message-----
> >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> >>> Sent: Thursday, February 7, 2013 5:42 PM
> >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> >>> Subject: Heads up - merge branch-trunk-win to trunk
> >>>
> >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> >>>The
> >>> goal was to make Hadoop natively integrated, full-featured, and
> >>>performance
> >>> and scalability tuned on Windows Server or Windows Azure.
> >>> We are happy to announce that a lot of progress has been made in this
> >>> regard.
> >>>
> >>> Initial work started in a feature branch, branch-1-win, based on
> >>>branch-1.
> >>> The details related to the work done in the branch can be seen in
> >>> CHANGES.txt<
> >>>
> >>>
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES.
> >>>branch-1-win.txt?view=markup
> >>> >.
> >>> This work has been ported to a branch, branch-trunk-win, based on
> trunk.
> >>> Merge patch for this is available on
> >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> >>> .
> >>>
> >>> Highlights of the work done so far:
> >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes
> >>> handle differences in platforms related to path names, process/task
> >>> management etc.
> >>> 2. Addition of winutils tools for managing file permissions and
> >>>ownership,
> >>> user group mapping, hardlinks, symbolic links, chmod, disk utilization,
> >>>and
> >>> process/task management.
> >>> 3. Added cmd scripts equivalent to existing shell scripts
> >>> hadoop-daemon.sh, start and stop scripts.
> >>> 4. Addition of block placement policy implemnation to support cloud
> >>> enviroment, more specifically Azure.
> >>>
> >>> We are very close to wrapping up the work in branch-trunk-win and
> >>>getting
> >>> ready for a merge. Currently the merge patch is passing close to 100%
> of
> >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> >>>into
> >>> trunk.
> >>>
> >>> Next steps:
> >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> >>> completes and precommit build is clean.
> >>> 2. Start a discussion on adding Jenkins precommit builds on windows and
> >>> how to integrate that with the existing commit process.
> >>>
> >>> Let me know if you have any questions.
> >>>
> >>> Regards,
> >>> Suresh
> >>>
> >>>
> >>
> >>
> >>--
> >>http://hortonworks.com/download/
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message