hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: [Vote] Merge branch-trunk-win to trunk
Date Wed, 27 Feb 2013 23:43:04 GMT
On Wed, Feb 27, 2013 at 2:54 PM, Suresh Srinivas <suresh@hortonworks.com>wrote:

> With that we need to decide how our precommit process looks.
> My inclination is to wait for +1 from precommit builds on
> both the platforms to ensure no issues are introduced.
> Thoughts?
>
> 2. Feature development impact
> Some questions have been raised about would new features
> need to be supported on both the platforms. Yes. I do not see a
> reason why features cannot work on both the platforms, with
> the exception of platform specific optimizations. This what Java
> gives us.
>
>
I'm concerned about the above. Personally, I don't have access to any
Windows boxes with development tools, and I know nothing about developing
on Windows. The only Windows I run is an 8GB VM with 1 GB RAM allocated,
for powerpoint :)

If I submit a patch and it gets -1 "tests failed" on the Windows slave, how
am I supposed to proceed?

I think a reasonable compromise would be that the tests should always
*build* on Windows before commit, and contributors should do their best to
look at the test logs for any Windows-specific failures. But, beyond
looking at the logs, a "-1 Tests failed on windows" should not block a
commit.

Those contributors who are interested in Windows being a first-class
platform should be responsible for watching the Windows builds and
debugging/fixing any regressions that might be Windows-specific.

I also think the KDE model that Harsh pointed out is an interesting one --
ie the idea that we would not merge windows support to trunk, but rather
treat is as a "parallel code line" which lives in the ASF and has its own
builds and releases. The windows team would periodically merge trunk->win
to pick up any new changes, and do a separate test/release process. I'm not
convinced this is the best idea, but worth discussion of pros and cons.

-Todd


>
> On Wed, Feb 27, 2013 at 11:56 AM, Eli Collins <eli@cloudera.com> wrote:
>
> > Bobby raises some good questions.  A related one, since most current
> > developers won't add Windows support for new features that are
> > platform specific is it assumed that Windows development will either
> > lag or will people actively work on keeping Windows up with the
> > latest?  And vice versa in case Windows support is implemented first.
> >
> > Is there a jira for resolving the outstanding TODOs in the code base
> > (similar to HDFS-2148)?  Looks like this merge doesn't introduce many
> > which is great (just did a quick diff and grep).
> >
> > Thanks,
> > Eli
> >
> > On Wed, Feb 27, 2013 at 8:17 AM, Robert Evans <evans@yahoo-inc.com>
> wrote:
> > > After this is merged in is Windows still going to be a second class
> > > citizen but happens to work for more than just development or is it a
> > > fully supported platform where if something breaks it can block a
> > release?
> > >  How do we as a community intend to keep Windows support from breaking?
> > > We don't have any Jenkins slaves to be able to run nightly tests to
> > > validate everything still compiles/runs.  This is not a blocker for me
> > > because we often rely on individuals and groups to test Hadoop, but I
> do
> > > think we need to have this discussion before we put it in.
> > >
> > > --Bobby
> > >
> > > On 2/26/13 4:55 PM, "Suresh Srinivas" <suresh@hortonworks.com> wrote:
> > >
> > >>I had posted heads up about merging branch-trunk-win to trunk on Feb
> 8th.
> > >>I
> > >>am happy to announce that we are ready for the merge.
> > >>
> > >>Here is a brief recap on the highlights of the work done:
> > >>- Command-line scripts for the Hadoop surface area
> > >>- Mapping the HDFS permissions model to Windows
> > >>- Abstracted and reconciled mismatches around differences in Path
> > >>semantics
> > >>in Java and Windows
> > >>- Native Task Controller for Windows
> > >>- Implementation of a Block Placement Policy to support cloud
> > >>environments,
> > >>more specifically Azure.
> > >>- Implementation of Hadoop native libraries for Windows (compression
> > >>codecs, native I/O)
> > >>- Several reliability issues, including race-conditions, intermittent
> > test
> > >>failures, resource leaks.
> > >>- Several new unit test cases written for the above changes
> > >>
> > >>Please find the details of the work in CHANGES.branch-trunk-win.txt -
> > >>Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<
> http://bit.ly/13QOSo9
> > >,
> > >>and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the
> work
> > >>ported from branch-1-win to a branch based on trunk.
> > >>
> > >>For details of the testing done, please see the thread -
> > >>http://bit.ly/WpavJ4. Merge patch for this is available on
> HADOOP-8562<
> > >>https://issues.apache.org/jira/browse/HADOOP-8562>.
> > >>
> > >>This was a large undertaking that involved developing code, testing the
> > >>entire Hadoop stack, including scale tests. This is made possible only
> > >>with
> > >>the contribution from many many folks in the community. Following
> people
> > >>contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
> Saha,
> > >>Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
> > Sumadhur
> > >>Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing Zhao,
> Thejas
> > >>Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan Kesavan, Ramya
> > >>Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo Nicholas Sze,
> > Suresh
> > >>Srinivas and Sanjay Radia. There are many others who contributed as
> well
> > >>providing feedback and comments on numerous jiras.
> > >>
> > >>The vote will run for seven days and will end on March 5, 6:00PM PST.
> > >>
> > >>Regards,
> > >>Suresh
> > >>
> > >>
> > >>
> > >>
> > >>On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
> > >><mahadv@microsoft.com>wrote:
> > >>
> > >>> It is super exciting to look at the prospect of these changes being
> > >>>merged
> > >>> to trunk. Having Windows as one of the supported Hadoop platforms is
> a
> > >>> fantastic opportunity both for the Hadoop project and Microsoft
> > >>>customers.
> > >>>
> > >>> This work began around a year back when a few of us started with a
> > basic
> > >>> port of Hadoop on Windows. Ever since, the Hadoop team in Microsoft
> > have
> > >>> made significant progress in the following areas:
> > >>> (PS: Some of these items are already included in Suresh's email, but
> > >>> including again for completeness)
> > >>>
> > >>> - Command-line scripts for the Hadoop surface area
> > >>> - Mapping the HDFS permissions model to Windows
> > >>> - Abstracted and reconciled mismatches around differences in Path
> > >>> semantics in Java and Windows
> > >>> - Native Task Controller for Windows
> > >>> - Implementation of a Block Placement Policy to support cloud
> > >>> environments, more specifically Azure.
> > >>> - Implementation of Hadoop native libraries for Windows (compression
> > >>> codecs, native I/O) - Several reliability issues, including
> > >>> race-conditions, intermittent test failures, resource leaks.
> > >>> - Several new unit test cases written for the above changes
> > >>>
> > >>> In the process, we have closely engaged with the Apache open source
> > >>> community and have got great support and assistance from the
> community
> > >>>in
> > >>> terms of contributing fixes, code review comments and commits.
> > >>>
> > >>> In addition, the Hadoop team at Microsoft has also made good progress
> > in
> > >>> other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many
> > >>>of
> > >>> these changes have already been committed to the respective trunks
> with
> > >>> help from various committers and contributors. It is great to see the
> > >>> commitment of the community to support multiple platforms, and we
> look
> > >>> forward to the day when a developer/customer is able to successfully
> > >>>deploy
> > >>> a complete solution stack based on Apache Hadoop releases.
> > >>>
> > >>> Next Steps:
> > >>>
> > >>> All of the above changes are part of the Windows Azure HDInsight and
> > >>> HDInsight Server products from Microsoft. We have successfully
> > >>>on-boarded
> > >>> several internal customers and have been running production workloads
> > on
> > >>> Windows Azure HDInsight. Our vision is to create a big data platform
> > >>>based
> > >>> on Hadoop, and we are committed to helping make Hadoop a world-class
> > >>> solution that anyone can use to solve their biggest data challenges.
> > >>>
> > >>> As an immediate next step, we would like to have a discussion around
> > how
> > >>> we can ensure that the quality of the mainline Hadoop branches on
> > >>>Windows
> > >>> is maintained. To this end, we would like to get to the state where
> we
> > >>>have
> > >>> pre-checkin validation gates and nightly test runs enabled on
> Windows.
> > >>>If
> > >>> you have any suggestions around this, please do send an email.  We
> are
> > >>> committed to helping sustain the long-term quality of Hadoop on both
> > >>>Linux
> > >>> and Windows.
> > >>>
> > >>> We sincerely thank the community for their contribution and support
> so
> > >>> far. And hope to continue having a close engagement in the future.
> > >>>
> > >>> -Microsoft HDInsight Team
> > >>>
> > >>>
> > >>> -----Original Message-----
> > >>> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> > >>> Sent: Thursday, February 7, 2013 5:42 PM
> > >>> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> > >>> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> > >>> Subject: Heads up - merge branch-trunk-win to trunk
> > >>>
> > >>> The support for Hadoop on Windows was proposed in HADOOP-8079<
> > >>> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year
> ago.
> > >>>The
> > >>> goal was to make Hadoop natively integrated, full-featured, and
> > >>>performance
> > >>> and scalability tuned on Windows Server or Windows Azure.
> > >>> We are happy to announce that a lot of progress has been made in this
> > >>> regard.
> > >>>
> > >>> Initial work started in a feature branch, branch-1-win, based on
> > >>>branch-1.
> > >>> The details related to the work done in the branch can be seen in
> > >>> CHANGES.txt<
> > >>>
> > >>>
> > http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANGES
> .
> > >>>branch-1-win.txt?view=markup
> > >>> >.
> > >>> This work has been ported to a branch, branch-trunk-win, based on
> > trunk.
> > >>> Merge patch for this is available on
> > >>> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> > >>> .
> > >>>
> > >>> Highlights of the work done so far:
> > >>> 1. Necessary changes in Hadoop to run natively on Windows. These
> > changes
> > >>> handle differences in platforms related to path names, process/task
> > >>> management etc.
> > >>> 2. Addition of winutils tools for managing file permissions and
> > >>>ownership,
> > >>> user group mapping, hardlinks, symbolic links, chmod, disk
> utilization,
> > >>>and
> > >>> process/task management.
> > >>> 3. Added cmd scripts equivalent to existing shell scripts
> > >>> hadoop-daemon.sh, start and stop scripts.
> > >>> 4. Addition of block placement policy implemnation to support cloud
> > >>> enviroment, more specifically Azure.
> > >>>
> > >>> We are very close to wrapping up the work in branch-trunk-win and
> > >>>getting
> > >>> ready for a merge. Currently the merge patch is passing close to 100%
> > of
> > >>> unit tests on Linux. Soon I will call for a vote to merge this branch
> > >>>into
> > >>> trunk.
> > >>>
> > >>> Next steps:
> > >>> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> > >>> completes and precommit build is clean.
> > >>> 2. Start a discussion on adding Jenkins precommit builds on windows
> and
> > >>> how to integrate that with the existing commit process.
> > >>>
> > >>> Let me know if you have any questions.
> > >>>
> > >>> Regards,
> > >>> Suresh
> > >>>
> > >>>
> > >>
> > >>
> > >>--
> > >>http://hortonworks.com/download/
> > >
> >
>
>
>
> --
> http://hortonworks.com/download/
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message