Return-Path: X-Original-To: apmail-hadoop-common-dev-archive@www.apache.org Delivered-To: apmail-hadoop-common-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 52326E456 for ; Sun, 3 Mar 2013 03:33:17 +0000 (UTC) Received: (qmail 15609 invoked by uid 500); 3 Mar 2013 03:33:15 -0000 Delivered-To: apmail-hadoop-common-dev-archive@hadoop.apache.org Received: (qmail 15524 invoked by uid 500); 3 Mar 2013 03:33:15 -0000 Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: common-dev@hadoop.apache.org Delivered-To: mailing list common-dev@hadoop.apache.org Received: (qmail 15473 invoked by uid 99); 3 Mar 2013 03:33:14 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Mar 2013 03:33:14 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of shv.hadoop@gmail.com designates 209.85.215.47 as permitted sender) Received: from [209.85.215.47] (HELO mail-la0-f47.google.com) (209.85.215.47) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 03 Mar 2013 03:33:08 +0000 Received: by mail-la0-f47.google.com with SMTP id fj20so3999790lab.20 for ; Sat, 02 Mar 2013 19:32:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=NRgTKbpJBwZ9MWsCySIYil0qX1kFpJh0pSekDcKEnhI=; b=Dblbt1l0so9n0SD1Yi8Xd05z5hGU/59LjpqiAutdnf+V50kxPpApnS3Dct2w9LzQdA UYwgehJUpPmDDBuzeTtzAA0vRZ/VhFP4BXXtf0hiTG2uKk9bTeITgVnpuiLHyaATAzz0 EyJNyZShGB8d7Xhj6pL9CuAjU231sfB3GjSLM2Cs0BckX6jwbBGIDhIDVDnwcDziFmrl 33qSizQIzFkdJkHwVIJxNCSZpQiEtdIG/zLni6em18InychA5lRd/Py7JNkL4Rib2Meg W612mY9ZHoGIFeYAkJDowC4DhzVNvkZxchdwvfHT+5a9PIfjMzCdj4K4WE/0rTH/mAmD 2JQA== MIME-Version: 1.0 X-Received: by 10.152.111.67 with SMTP id ig3mr13691512lab.41.1362281566194; Sat, 02 Mar 2013 19:32:46 -0800 (PST) Received: by 10.152.109.78 with HTTP; Sat, 2 Mar 2013 19:32:45 -0800 (PST) In-Reply-To: References:

Date: Sat, 2 Mar 2013 19:32:45 -0800 Message-ID: Subject: Re: [Vote] Merge branch-trunk-win to trunk From: Konstantin Shvachko To: Matt Foley Cc: "common-dev@hadoop.apache.org" , "hdfs-dev@hadoop.apache.org" , mapreduce-dev@hadoop.apache.org, yarn-dev@hadoop.apache.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Checked: Checked by ClamAV on apache.org Hi Matt, On Sat, Mar 2, 2013 at 12:32 PM, Matt Foley wrote: > Konstantin, > I would like to explore what it would take to remove this perceived > impediment -- Glad you decided to explore. Thank you. > although I reserve the right to argue that this is not > pre-requisite to merging the cross-platform support patch. It's your right indeed. So as mine to question what the platform support means for you, which I believe remained unclear. I do not impede the change as you should have noticed. My requirement comes from my perception of the support, which means to me exactly two things: 1. The ability to recognise the code is broken for the platform 2. The ability to test new patches on the platform The latter is problematic, as many noticed in this thread, for those whose customary environment does not include Windows. > If we implemented full "test-patch" support for Windows on trunk, would that > fulfill both your items #1 and #2? Please note that: > a) Pushing the "Patch Available" button in Jira shall cause a pre-commit > build to start within, I believe, 20 minutes. > b) That build keeps logs for both java build and unit tests for several > days, that are accessible to all viewers. In item #1 I mostly asking for the nightly build, which is simpler than "test-patch". The latter would be ideal from the platform support viewpoint, but it is for the community to decide if we want to add extra +3 hours to the build. Nightly build in my understanding is triggered by the timer rather than by Jira's "submit patch" button. On Jenkins build configuration you can specify it under "Build periodically". > So, does this provide sufficient on-demand support that we don't have to > implement a whole new on-demand VM support structure of some sort for #2 > (which would be an extraordinary and impractical demand)? I did not mention VMs. Item #2 means a build, which runs "test-patch" target with the file specified by a user (instead of a jira attachment). When user clicks "Build Now" link a box is displayed where the user can enter the file path containing the patch. This can be specified in the Build Configuration under "This build is parameterized" by choosing AddParameter / FileParameter. The build can run on the same Windows machine as the nightly build. Such build will let people test their patches for Windows on Jenkins if they don't posses a license for the right version of Windows. I hope this will not turn into extraordinary or impractical effort. Thanks, --Konst > Thanks, > --Matt > > > On Fri, Mar 1, 2013 at 12:18 PM, Konstantin Shvachko > wrote: >> >> -1 >> We should have a CI infrastructure in place before we can commit to >> supporting Windows platform. >> >> Eric is right Win/Cygwin was supported since day one. >> I had a Windows box under my desk running nightly builds back in 2006-07. >> People were irritated but I was filing windows bugs until 0.22 release. >> Times changing and I am glad to see wider support for Win platform. >> >> But in order to make it work you guys need to put the CI process in place >> >> 1. windows jenkins build: could be nightly or PreCommit. >> - Nightly would mean that changes can be committed to trunk based on >> linux PreCommit build. And people will file bugs if the change broke >> Windows nightly build. >> - PreCommit-win build will mean automatic reporting failed tests to >> respective jira blocking commits the same way as it is now with linux >> PreCommit builds. >> We should discuss which way is more efficient for developers. >> >> 2. On-demand-windows Jenkins build. >> I see it as a build to which I can attach my patch and the build will >> run my changes on a dedicated windows box. >> That way people can test their changes without having personal windows >> nodes. >> >> I think this is the minimal set of requirement for us to be able to >> commit to the new platform. >> Right now I see only one windows related build >> https://builds.apache.org/view/Hadoop/job/Hadoop-1-win/ >> Which was failing since Sept 8, 2012 and did not run in the last month. >> >> Thanks, >> --Konst >> >> On Thu, Feb 28, 2013 at 8:47 PM, Eric Baldeschwieler >> wrote: >> > +1 (non-binding) >> > >> > A few of observations: >> > >> > - Windows has actually been a supported platform for Hadoop since 0.1 . >> > Doug championed supporting windows then and we've continued to do it with >> > varying vigor over time. To my knowledge we've never made a decision to >> > drop windows support. The change here is improving our support and dropping >> > the requirement of cigwin. We had Nutch windows users on the list in 2006 >> > and we've been supporting windows FS requirements since inception. >> > >> > - A little pragmatism will go a long way. As a community we've got to >> > stay committed to keeping hadoop simple (so it does work on many platforms) >> > and extending it to take advantage of key emerging OS/hardware features, >> > such as containers, new FSs, virtualization, flash ... We should all plan >> > to let new features & optimizations emerge that don't work everywhere, if >> > they are compelling and central to hadoop's mission of being THE best fabric >> > for storing and processing big data. >> > >> > - A UI project like KDE has to deal with the MANY differences between >> > windows and linux UI APIs. Hadoop faces no such complex challenge and hence >> > can be maintained from a single codeline IMO. It is mostly abstracted from >> > the OS APIs via Java and our design choices. Where it is not we can >> > continue to add plugable abstractions. >> > > >