yetus-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@effectivemachines.com>
Subject Re: Avro Pre-commit.
Date Tue, 13 Sep 2016 13:13:40 GMT

I've been thinking about this issue.  With a bit of work, I think the approach should probably
be to:

	a) make a custom build tool that knows which dirs use which other tools
	b) add a new file in each dir that can tell precommit where to break for modules

For example, let's say we have a dir structure like this:

		build.sh
		c/
			Makefile
		java/
			pom.xml
		perl/
			Makefile.PL

Let's add a file called multibuild.sh that precommit can use:

		build.sh
		multibuild.sh
		c/
			Makefile
			multibuild.sh
		java/
			pom.xml
			multibuild.sh
		perl/
			Makefile.PL
			multibuild.sh


In our "multibuild" build tool plugin, we'd define

		function multibuild_buildfile
		{
			echo "multibuild.sh"
		}

	Now when precommit gets a patch, it will know that a change in c/ is a different module than
a change in java/ or perl/ because of the presence of the multibuild.sh file.  This prevents
the "build the world" problem for a largely diverse project like avro. 

	The place where this gets hard is our theoretical multibuild_executor function because precommit
doesn't pass as a param the module that's currently being processed. (we should probably fix
that) But we can still make it work and reality, it might have led us down a false path. 
If we make this function look like this:

		function multibuild_executor
		{
			echo "multibuild.sh"
		}

and make sure that BUILDTOOLCWD=module (the default), then precommit will run our file that
we placed to actually do the work.  This file can then take whatever parameters we're getting
passed and convert it as appropriate to the build tool for that directory. Since this is a
custom build tool, you can dictate what parameters to pass to which modules via _modules_workers.
 For example, an easy route might be:

		function multibuild_modules_workers
		{
			modules_workers $2
		}

then multibuild.sh for each language becomes a series of case statements.  For C, it might
look like:

		testname=$1
		shift

		case ${testname} in
			compile)
				make -Dcustomflag=1
			;;
			doc)
				make man
			unit)
				make -Danotherflag=1 test
			;;
			*)
			;;
		esac

	It's pretty hacky, but theoretically it would work until we get a chance to make precommit
smarter about multiple build tools in a single project. (which would be a pretty hard undertaking,
but something we should probably do)


> On Sep 10, 2016, at 6:32 PM, suraj acharya <suraj.spa@gmail.com> wrote:
> 
> Yes.
> The ./build.sh test
> <https://github.com/apache/avro/blob/master/build.sh#L40> command helps run
> all the tests from the top level.
> However, I am not sure if we need to run all the tests when a change is
> made?
> I was thinking of running language specific tests for changes in the
> specific directory.
> For example for changes in python will trigger ant test
> <https://github.com/apache/avro/blob/master/build.sh#L43>.
> If this is not a good idea let me know. Ill start another tread on dev@avro
> to see what the community thinks about it
> 
> Thanks
> 
> -Suraj Acharya
> 
> On Sat, Sep 10, 2016 at 5:45 PM, Sean Busbey <busbey@apache.org> wrote:
> 
>> my understanding of Avro's build system is that there are per-language
>> build tools and everything gets unified via shell scripts. is there
>> just the top level build.sh file, or do the individual components have
>> their own?
>> 
>> On Sat, Sep 10, 2016 at 11:48 AM, suraj acharya <suraj.spa@gmail.com>
>> wrote:
>>> Hi,
>>> I have been working on AVRO-1887
>>> <https://issues.apache.org/jira/browse/AVRO-1887> to add pre-commit
>> test to
>>> avro.
>>> I have been successful in adding the personality for java file using
>> maven.
>>> I having some questions about the other plugins.
>>> Example :ant
>>> 
>>> The structure of avro is that there is a top level pom.
>>> And then there is different langs in subdirectory below it.
>>> So there is avro/lang/py
>>> <https://github.com/apache/avro/tree/master/lang/py>which has python
>> code
>>> in it. The build tool for that is ant.
>>> My understanding is running add_test ant will run ant test.
>>> However, that doesn't actually run any tests. My assumption is that since
>>> there is no build.xml present there it is unable to run any ant tests.
>>> 
>>> I tried cd to lang/py and run add_test ant from there. However, it throws
>>> an error saying unable to find pom.xml.
>>> 
>>> Can someone give me some pointers on how to continue?
>>> 
>>> 
>>> Thanks
>>> -Suraj Acharya
>> 


Mime
View raw message