Mailing-List: contact dev-help@yetus.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@yetus.apache.org
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\))
Subject: Re: Avro Pre-commit.
From: Allen Wittenauer <aw@effectivemachines.com>
In-Reply-To: <CAFK5+RKCQc-raz7G5+CR8BwVCzZu2g7Fs=xvECd_=AuxVCVByQ@mail.gmail.com>
Date: Tue, 13 Sep 2016 06:13:40 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <60E9D2CC-B77B-4B56-AF6F-EE4F11CF43E7@effectivemachines.com>
References: <CAFK5+RL+3Z8JJmT+y2ZkMbDv2QuZti4zv-D_Birb484aWRXMJQ@mail.gmail.com> <CAN5cbe5JxNT3ajUfOAjLgJvw7GUW=C7G1nu8T-99V8oyykd_Ew@mail.gmail.com> <CAFK5+RKCQc-raz7G5+CR8BwVCzZu2g7Fs=xvECd_=AuxVCVByQ@mail.gmail.com>
To: dev@yetus.apache.org
archived-at: Tue, 13 Sep 2016 13:13:56 -0000


I've been thinking about this issue.  With a bit of work, I think the =
approach should probably be to:

	a) make a custom build tool that knows which dirs use which =
other tools
	b) add a new file in each dir that can tell precommit where to =
break for modules

For example, let's say we have a dir structure like this:

		build.sh
		c/
			Makefile
		java/
			pom.xml
		perl/
			Makefile.PL

Let's add a file called multibuild.sh that precommit can use:

		build.sh
		multibuild.sh
		c/
			Makefile
			multibuild.sh
		java/
			pom.xml
			multibuild.sh
		perl/
			Makefile.PL
			multibuild.sh


In our "multibuild" build tool plugin, we'd define

		function multibuild_buildfile
		{
			echo "multibuild.sh"
		}

	Now when precommit gets a patch, it will know that a change in =
c/ is a different module than a change in java/ or perl/ because of the =
presence of the multibuild.sh file.  This prevents the "build the world" =
problem for a largely diverse project like avro.=20

	The place where this gets hard is our theoretical =
multibuild_executor function because precommit doesn't pass as a param =
the module that's currently being processed. (we should probably fix =
that) But we can still make it work and reality, it might have led us =
down a false path.  If we make this function look like this:

		function multibuild_executor
		{
			echo "multibuild.sh"
		}

and make sure that BUILDTOOLCWD=3Dmodule (the default), then precommit =
will run our file that we placed to actually do the work.  This file can =
then take whatever parameters we're getting passed and convert it as =
appropriate to the build tool for that directory. Since this is a custom =
build tool, you can dictate what parameters to pass to which modules via =
_modules_workers.  For example, an easy route might be:

		function multibuild_modules_workers
		{
			modules_workers $2
		}

then multibuild.sh for each language becomes a series of case =
statements.  For C, it might look like:

		testname=3D$1
		shift

		case ${testname} in
			compile)
				make -Dcustomflag=3D1
			;;
			doc)
				make man
			unit)
				make -Danotherflag=3D1 test
			;;
			*)
			;;
		esac

	It's pretty hacky, but theoretically it would work until we get =
a chance to make precommit smarter about multiple build tools in a =
single project. (which would be a pretty hard undertaking, but something =
we should probably do)


> On Sep 10, 2016, at 6:32 PM, suraj acharya <suraj.spa@gmail.com> =
wrote:
>=20
> Yes.
> The ./build.sh test
> <https://github.com/apache/avro/blob/master/build.sh#L40> command =
helps run
> all the tests from the top level.
> However, I am not sure if we need to run all the tests when a change =
is
> made?
> I was thinking of running language specific tests for changes in the
> specific directory.
> For example for changes in python will trigger ant test
> <https://github.com/apache/avro/blob/master/build.sh#L43>.
> If this is not a good idea let me know. Ill start another tread on =
dev@avro
> to see what the community thinks about it
>=20
> Thanks
>=20
> -Suraj Acharya
>=20
> On Sat, Sep 10, 2016 at 5:45 PM, Sean Busbey <busbey@apache.org> =
wrote:
>=20
>> my understanding of Avro's build system is that there are =
per-language
>> build tools and everything gets unified via shell scripts. is there
>> just the top level build.sh file, or do the individual components =
have
>> their own?
>>=20
>> On Sat, Sep 10, 2016 at 11:48 AM, suraj acharya <suraj.spa@gmail.com>
>> wrote:
>>> Hi,
>>> I have been working on AVRO-1887
>>> <https://issues.apache.org/jira/browse/AVRO-1887> to add pre-commit
>> test to
>>> avro.
>>> I have been successful in adding the personality for java file using
>> maven.
>>> I having some questions about the other plugins.
>>> Example :ant
>>>=20
>>> The structure of avro is that there is a top level pom.
>>> And then there is different langs in subdirectory below it.
>>> So there is avro/lang/py
>>> <https://github.com/apache/avro/tree/master/lang/py>which has python
>> code
>>> in it. The build tool for that is ant.
>>> My understanding is running add_test ant will run ant test.
>>> However, that doesn't actually run any tests. My assumption is that =
since
>>> there is no build.xml present there it is unable to run any ant =
tests.
>>>=20
>>> I tried cd to lang/py and run add_test ant from there. However, it =
throws
>>> an error saying unable to find pom.xml.
>>>=20
>>> Can someone give me some pointers on how to continue?
>>>=20
>>>=20
>>> Thanks
>>> -Suraj Acharya
>>=20