hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bikas Saha <bi...@hortonworks.com>
Subject RE: [Vote] Merge branch-trunk-win to trunk
Date Fri, 01 Mar 2013 17:55:17 GMT
That's sounds like a reasonable approach. Like you say below, we need to
ensure is that the Java side of OS specific optimizations is generic and
wont need drastic surgery when that optimization is ported to another
platform.

Bikas

-----Original Message-----
From: Uma Maheswara Rao G [mailto:maheswara@huawei.com]
Sent: Thursday, February 28, 2013 9:20 PM
To: hdfs-dev@hadoop.apache.org; common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
Subject: RE: [Vote] Merge branch-trunk-win to trunk

+1 (non-binding)

 Thanks a lot for the work done by Suresh and team of community!
I don't think there will be much problems because of platform dependancy
as our development is with Java. If we have some native code portings,
then dev members has to take care of them.
One question regarding to it:
Ex: if one contributor is giving the patch for some native code porting
for some performance improvements and he is interested only in Linux. Then
I hope some other contributors will help in getting the windows patch if
possible.
If others busy to get that done with in time lines, then I think we can
commit Linux support patch and leave one JIRA open for Windows support? [
make sure that porting options configurable and platform check and give
release note about platform support..etc?]

Regards,
Uma

________________________________________
From: Suresh Srinivas [suresh@hortonworks.com]
Sent: Wednesday, February 27, 2013 4:25 AM
To: common-dev@hadoop.apache.org
Cc: yarn-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org;
mapreduce-dev@hadoop.apache.org
Subject: [Vote] Merge branch-trunk-win to trunk

I had posted heads up about merging branch-trunk-win to trunk on Feb 8th.
I am happy to announce that we are ready for the merge.

Here is a brief recap on the highlights of the work done:
- Command-line scripts for the Hadoop surface area
- Mapping the HDFS permissions model to Windows
- Abstracted and reconciled mismatches around differences in Path
semantics in Java and Windows
- Native Task Controller for Windows
- Implementation of a Block Placement Policy to support cloud
environments, more specifically Azure.
- Implementation of Hadoop native libraries for Windows (compression
codecs, native I/O)
- Several reliability issues, including race-conditions, intermittent test
failures, resource leaks.
- Several new unit test cases written for the above changes

Please find the details of the work in CHANGES.branch-trunk-win.txt -
Common changes<http://bit.ly/Xe7Ynv>, HDFS changes<http://bit.ly/13QOSo9>,
and YARN and MapReduce changes <http://bit.ly/128zzMt>. This is the work
ported from branch-1-win to a branch based on trunk.

For details of the testing done, please see the thread -
http://bit.ly/WpavJ4. Merge patch for this is available on HADOOP-8562<
https://issues.apache.org/jira/browse/HADOOP-8562>.

This was a large undertaking that involved developing code, testing the
entire Hadoop stack, including scale tests. This is made possible only
with the contribution from many many folks in the community. Following
people contributed to this work: Ivan Mitic, Chuan Liu, Ramya Sunil, Bikas
Saha, Kanna Karanam, John Gordon, Brandon Li, Chris Nauroth, David Lao,
Sumadhur Reddy Bolli, Arpit Agarwal, Ahmed El Baz, Mike Liddell, Jing
Zhao, Thejas Nair, Steve Maine, Ganeshan Iyer, Raja Aluri, Giridharan
Kesavan, Ramya Bharathi Nimmagadda, Daryn Sharp, Arun Murthy, Tsz-Wo
Nicholas Sze, Suresh Srinivas and Sanjay Radia. There are many others who
contributed as well providing feedback and comments on numerous jiras.

The vote will run for seven days and will end on March 5, 6:00PM PST.

Regards,
Suresh




On Thu, Feb 7, 2013 at 6:41 PM, Mahadevan Venkatraman
<mahadv@microsoft.com>wrote:

> It is super exciting to look at the prospect of these changes being
> merged to trunk. Having Windows as one of the supported Hadoop
> platforms is a fantastic opportunity both for the Hadoop project and
Microsoft customers.
>
> This work began around a year back when a few of us started with a
> basic port of Hadoop on Windows. Ever since, the Hadoop team in
> Microsoft have made significant progress in the following areas:
> (PS: Some of these items are already included in Suresh's email, but
> including again for completeness)
>
> - Command-line scripts for the Hadoop surface area
> - Mapping the HDFS permissions model to Windows
> - Abstracted and reconciled mismatches around differences in Path
> semantics in Java and Windows
> - Native Task Controller for Windows
> - Implementation of a Block Placement Policy to support cloud
> environments, more specifically Azure.
> - Implementation of Hadoop native libraries for Windows (compression
> codecs, native I/O) - Several reliability issues, including
> race-conditions, intermittent test failures, resource leaks.
> - Several new unit test cases written for the above changes
>
> In the process, we have closely engaged with the Apache open source
> community and have got great support and assistance from the community
> in terms of contributing fixes, code review comments and commits.
>
> In addition, the Hadoop team at Microsoft has also made good progress
> in other projects including Hive, Pig, Sqoop, Oozie, HCat and HBase.
> Many of these changes have already been committed to the respective
> trunks with help from various committers and contributors. It is great
> to see the commitment of the community to support multiple platforms,
> and we look forward to the day when a developer/customer is able to
> successfully deploy a complete solution stack based on Apache Hadoop
releases.
>
> Next Steps:
>
> All of the above changes are part of the Windows Azure HDInsight and
> HDInsight Server products from Microsoft. We have successfully
> on-boarded several internal customers and have been running production
> workloads on Windows Azure HDInsight. Our vision is to create a big
> data platform based on Hadoop, and we are committed to helping make
> Hadoop a world-class solution that anyone can use to solve their biggest
data challenges.
>
> As an immediate next step, we would like to have a discussion around
> how we can ensure that the quality of the mainline Hadoop branches on
> Windows is maintained. To this end, we would like to get to the state
> where we have pre-checkin validation gates and nightly test runs
> enabled on Windows. If you have any suggestions around this, please do
> send an email.  We are committed to helping sustain the long-term
> quality of Hadoop on both Linux and Windows.
>
> We sincerely thank the community for their contribution and support so
> far. And hope to continue having a close engagement in the future.
>
> -Microsoft HDInsight Team
>
>
> -----Original Message-----
> From: Suresh Srinivas [mailto:suresh@hortonworks.com]
> Sent: Thursday, February 7, 2013 5:42 PM
> To: common-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org;
> hdfs-dev@hadoop.apache.org; mapreduce-dev@hadoop.apache.org
> Subject: Heads up - merge branch-trunk-win to trunk
>
> The support for Hadoop on Windows was proposed in HADOOP-8079<
> https://issues.apache.org/jira/browse/HADOOP-8079> almost a year ago.
> The goal was to make Hadoop natively integrated, full-featured, and
> performance and scalability tuned on Windows Server or Windows Azure.
> We are happy to announce that a lot of progress has been made in this
> regard.
>
> Initial work started in a feature branch, branch-1-win, based on
branch-1.
> The details related to the work done in the branch can be seen in
> CHANGES.txt<
> http://svn.apache.org/viewvc/hadoop/common/branches/branch-1-win/CHANG
> ES.branch-1-win.txt?view=markup
> >.
> This work has been ported to a branch, branch-trunk-win, based on trunk.
> Merge patch for this is available on
> HADOOP-8562<https://issues.apache.org/jira/browse/HADOOP-8562>
> .
>
> Highlights of the work done so far:
> 1. Necessary changes in Hadoop to run natively on Windows. These
> changes handle differences in platforms related to path names,
> process/task management etc.
> 2. Addition of winutils tools for managing file permissions and
> ownership, user group mapping, hardlinks, symbolic links, chmod, disk
> utilization, and process/task management.
> 3. Added cmd scripts equivalent to existing shell scripts
> hadoop-daemon.sh, start and stop scripts.
> 4. Addition of block placement policy implemnation to support cloud
> enviroment, more specifically Azure.
>
> We are very close to wrapping up the work in branch-trunk-win and
> getting ready for a merge. Currently the merge patch is passing close
> to 100% of unit tests on Linux. Soon I will call for a vote to merge
> this branch into trunk.
>
> Next steps:
> 1. Call for vote to merge branch-trunk-win to trunk, when the work
> completes and precommit build is clean.
> 2. Start a discussion on adding Jenkins precommit builds on windows
> and how to integrate that with the existing commit process.
>
> Let me know if you have any questions.
>
> Regards,
> Suresh
>
>


--
http://hortonworks.com/download/

Mime
View raw message