hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <tlip...@gmail.com>
Subject Re: Developing cross-component patches post-split
Date Thu, 02 Jul 2009 05:13:55 GMT
On Wed, Jul 1, 2009 at 10:11 PM, Dhruba Borthakur <dhruba@gmail.com> wrote:

> Hi Todd,
>
> Another option (one that is used by Hive) is to have an ant macro that can
> be overridden from the ant command line. This macro points to the location
> of the common.jar. By default, it is set to the same value as it is now. If
> a developer has a common jar that is built in his/her directory, he/she can
> set this macro from the command line while compiling hdfs.
>
> For example,
>  ant test
> does the same as it does now, but
>  ant -Dhadoop.common.jar=/home/dhruba/common/hadoop-common.jar test will
> pick up the common jar from my home directory.
>
> is this feasible?
>

That's feasible, but it will still require having a built jar in one or
another repository after every new commit (yuck!) I imagine in hive's case
it's reasonably rare that you have to import a new hadoop dev jar in, since
you mostly target existing stable releases. This is going to be happening
all the time in MR/HDFS, at least for the forseeable future imho.

-Todd

On Wed, Jul 1, 2009 at 6:45 PM, Todd Lipcon <tlipcon@gmail.com> wrote:

> On Wed, Jul 1, 2009 at 2:10 PM, Philip Zeyliger <philip@cloudera.com>
> wrote:
>
> > -1 to checking in jars.  It's quite a bit of bloat in the repository
> (which
> > admittedly affects the git.apache folks more than the svn folks), but
> it's
> > also cumbersome to develop.
> >
> > It'd be nice to have a one-liner that builds the equivalent of the
> tarball
> > built by "ant binary" in the old world.  When you're working on
something
> > that affects both common and hdfs, it'll be pretty painful to make the
> jars
> > in common, move them over to hdfs, and then compile hdfs.
> >
> > Could the build.xml in hdfs call into common's build.xml and build
common
> > as
> > part of building hdfs?  Or perhaps have a separate "top-level" build
file
> > that builds everything?
> >
>
> Agree with Phillip here. Requiring a new jar to be checked in anywhere
> after
> every common commit seems unscalable and nonperformant. For git users this
> will make the repository size baloon like crazy (the jar is 400KB and we
> have around 5300 commits so far = 2GB!). For svn users it will still mean
> that every "svn update" requires a download of a new jar. Using svn
> externals to manage them also complicates things when trying to work on a
> cross-component patch with two dirty directories - you really need a
> symlink
> between your working directories rather than through the SVN tree.
>
> I think it would be reasonable to require that developers check out a
> structure like:
>
> working-dir/
>  hadoop-common/
>  hadoop-mapred/
>  hadoop-hdfs/
>
> We can then use relative paths for the mapred->common and hdfs->common
> dependencies. Those who only work on HDFS or only work on mapred will not
> have to check out the other, but everyone will check out common.
>
> Whether there exists a fourth repository (eg hadoop-build) that has a
> build.xml that ties together the other build.xmls is another open question
> IMO.
>
> -Todd
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message