Mailing-List: contact common-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-dev@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of dhruba@gmail.com designates
 209.85.221.188 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type;
        b=RA9rZSp2aweQZ+qtEOGPSGjiiDtP286tZsKnYc9jd7PwiB/LhosplBnARgVfOJLtMS
         IkScfIIjW6RRjHUFn7B3P1cP82SvdrGE7VGU7yJgnJj7ydQF9jwkRfIHz47lH1JrWMK2
         Pj3+PZqNY/Y8KjtznE6tSd0ZWMZxmlyPj11Bc=
MIME-Version: 1.0
In-Reply-To: <a8b6b1b40907011845q796a3d7due056ac1ac3c34836@mail.gmail.com>
References: <45f85f70907011344t5e2e4fd8nf31800aa30797ac7@mail.gmail.com>
	 <4A4BCCEA.3000604@apache.org>
	 <15da8a100907011410l2ad4c4afq5b05a488240433cf@mail.gmail.com>
	 <a8b6b1b40907011845q796a3d7due056ac1ac3c34836@mail.gmail.com>
Date: Wed, 1 Jul 2009 22:11:05 -0700
Message-ID: <4aa34eb70907012211r44011115w8bf566b9ab23de12@mail.gmail.com>
Subject: Re: Developing cross-component patches post-split
From: Dhruba Borthakur <dhruba@gmail.com>
To: common-dev@hadoop.apache.org
Content-Type: multipart/alternative; boundary=0015175cb5321b16ee046db211ca

--0015175cb5321b16ee046db211ca
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit

Hi Todd,

Another option (one that is used by Hive) is to have an ant macro that can
be overridden from the ant command line. This macro points to the location
of the common.jar. By default, it is set to the same value as it is now. If
a developer has a common jar that is built in his/her directory, he/she can
set this macro from the command line while compiling hdfs.

For example,
  ant test
does the same as it does now, but
  ant -Dhadoop.common.jar=/home/dhruba/common/hadoop-common.jar test will
pick up the common jar from my home directory.

is this feasible?

thanks,
dhruba


On Wed, Jul 1, 2009 at 6:45 PM, Todd Lipcon <tlipcon@gmail.com> wrote:

> On Wed, Jul 1, 2009 at 2:10 PM, Philip Zeyliger <philip@cloudera.com>
> wrote:
>
> > -1 to checking in jars.  It's quite a bit of bloat in the repository
> (which
> > admittedly affects the git.apache folks more than the svn folks), but
> it's
> > also cumbersome to develop.
> >
> > It'd be nice to have a one-liner that builds the equivalent of the
> tarball
> > built by "ant binary" in the old world.  When you're working on something
> > that affects both common and hdfs, it'll be pretty painful to make the
> jars
> > in common, move them over to hdfs, and then compile hdfs.
> >
> > Could the build.xml in hdfs call into common's build.xml and build common
> > as
> > part of building hdfs?  Or perhaps have a separate "top-level" build file
> > that builds everything?
> >
>
> Agree with Phillip here. Requiring a new jar to be checked in anywhere
> after
> every common commit seems unscalable and nonperformant. For git users this
> will make the repository size baloon like crazy (the jar is 400KB and we
> have around 5300 commits so far = 2GB!). For svn users it will still mean
> that every "svn update" requires a download of a new jar. Using svn
> externals to manage them also complicates things when trying to work on a
> cross-component patch with two dirty directories - you really need a
> symlink
> between your working directories rather than through the SVN tree.
>
> I think it would be reasonable to require that developers check out a
> structure like:
>
> working-dir/
>  hadoop-common/
>  hadoop-mapred/
>  hadoop-hdfs/
>
> We can then use relative paths for the mapred->common and hdfs->common
> dependencies. Those who only work on HDFS or only work on mapred will not
> have to check out the other, but everyone will check out common.
>
> Whether there exists a fourth repository (eg hadoop-build) that has a
> build.xml that ties together the other build.xmls is another open question
> IMO.
>
> -Todd
>

--0015175cb5321b16ee046db211ca--