From general-return-5247-apmail-hadoop-general-archive=hadoop.apache.org@hadoop.apache.org Thu Jul 26 20:58:36 2012 Return-Path: X-Original-To: apmail-hadoop-general-archive@minotaur.apache.org Delivered-To: apmail-hadoop-general-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id D89C99A97 for ; Thu, 26 Jul 2012 20:58:36 +0000 (UTC) Received: (qmail 61019 invoked by uid 500); 26 Jul 2012 20:58:35 -0000 Delivered-To: apmail-hadoop-general-archive@hadoop.apache.org Received: (qmail 60945 invoked by uid 500); 26 Jul 2012 20:58:35 -0000 Mailing-List: contact general-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: general@hadoop.apache.org Delivered-To: mailing list general@hadoop.apache.org Received: (qmail 60936 invoked by uid 99); 26 Jul 2012 20:58:35 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jul 2012 20:58:35 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [209.85.160.48] (HELO mail-pb0-f48.google.com) (209.85.160.48) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 26 Jul 2012 20:58:25 +0000 Received: by pbbrq8 with SMTP id rq8so4388173pbb.35 for ; Thu, 26 Jul 2012 13:58:04 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=zxCw7ZcO87H9FuhDnzwjOvzkz18S/b0dbetddKhBcvA=; b=lYdesrn57vYgEzX6GU401tqraeDohNdC5WGU77Fz+5GqfYQl26E5VDEUO387N3IAhX hAkvgFFBOrRct3w3vD2HGdWCIVeuh69Jl0gBTGGNVUR3Aug68TGa4tYTDSYYFh2kiGRh V5iBfVEKMNNFTpI7c5cIpqn++FZ5s0jAmxy02g6mAcYVYCjO1z9CpP8CZnwdDzosaLII nqXi/Gs52/oDZmDAH2plIgDqBUErfvI8fdG5Ud+Fj8gg78e+pzX8lBQRe0e4Cstvt/D+ 0iBEMfNpOYzrXXtP2D+VihpCZWiNy8eJVPEKwkHz0OSinLMPX5IAhiTD3kaC8/DbxOYz +kMQ== Received: by 10.68.231.39 with SMTP id td7mr8354133pbc.3.1343336284598; Thu, 26 Jul 2012 13:58:04 -0700 (PDT) Received: from [10.10.11.42] (host1.hortonworks.com. [70.35.59.2]) by mx.google.com with ESMTPS id gh9sm392755pbc.20.2012.07.26.13.58.03 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 26 Jul 2012 13:58:03 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1084) Subject: Re: [DISCUSS] - YARN as a sub-project of Apache Hadoop From: Hitesh Shah In-Reply-To: Date: Thu, 26 Jul 2012 13:58:02 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: general@hadoop.apache.org X-Mailer: Apple Mail (2.1084) X-Gm-Message-State: ALoCoQnfB5Vco265LANvz95CeCNlV4VTD2xMuA3jOtgSopUaziGJmSgLLQ11s19V/v3oMnno+HWi +1. -- Hitesh On Jul 25, 2012, at 6:40 PM, Arun C Murthy wrote: > Folks, >=20 > It's been nearly a year since we merged Hadoop YARN into trunk and we = have made several releases since. >=20 > It's exciting to see various open-source communities (both in the ASF = and externally) start to explore integration with YARN such as Apache = Hama, Apache Giraph, Apache S4, Spark etc. This promises to help us = realize our hopes of making Apache Hadoop a much more general data = processing platform (& storage, of course) and not tied to MapReduce = alone for processing data. Furthermore, we already have people = contributing interesting prototypes such as DistributedShell and PaaS on = YARN. >=20 > Given this, I think it would be useful to make YARN a sub-project of = Apache Hadoop along with Common, HDFS & MapReduce. I believe this would = help other communities realize that they could consider using YARN as a = general-purpose resource management layer and help us enhance YARN = beyond it's humble beginnings.=20 >=20 > Clearly, YARN and MapReduce are different enough that they can and = will attract a diverse community. >=20 > I'd like to clarify that this proposal *does not* mean we move the = code base out of hadoop/common/ tree. It just alleviates hadoop-yarn = alongside hadoop-common, hadoop-hdfs & hadoop-mapreduce in hadoop/trunk. = Also, there would be *no changes* to release cycles - YARN would be = co-released with Common, HDFS & MapReduce. >=20 > Thoughts? >=20 > ---- >=20 > What does it mean to the Hadoop developer community? >=20 > # Project dependencies >=20 > The change is that Hadoop would now have 4 sub-projects: Common, HDFS, = YARN & MapReduce. As today, the dependencies *do not change*:=20 > - Common is the base > - HDFS depends only on Common > - YARN depends only on Common & HDFS=20 > - MapReduce depends on Common, HDFS & YARN. >=20 > # Jira & Mailing lists >=20 > We would have a separate YARN jira project and a yarn-dev@ mailing = list. >=20 > We already use separate MAPREDUCE jira issues for making changes to = YARN (ResourceManager, NodeManager) and to the MapReduce framework = (MapReduce ApplicationMaster, MapReduce runtime etc.). Hence, this isn't = a much of a change. >=20 > # Subversion >=20 > Not much at all! YARN has, since the beginning, been developed with = the understanding that it is very independent of MapReduce and the = code-bases are already independent i.e. = hadoop-mapreduce-project/hadoop-yarn and = hadoop-mapreduce-project/hadoop-mapreduce-client.=20 >=20 > Essentially the change would be: > $ svn mv hadoop-mapreduce-project/hadoop-yarn = hadoop-yarn-project/hadoop-yarn > ... and the necessary, albeit small, changes to our maven build = infrastructure. >=20 > # Release Cycles >=20 > No changes. >=20 > YARN would be co-released with Common, HDFS & MapReduce, as is the = case today. >=20 > thanks, > Arun