lucy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Robert Muir <rcm...@gmail.com>
Subject Re: [lucy-dev] Parallel compilation
Date Tue, 16 Nov 2010 12:51:43 GMT
On Mon, Nov 15, 2010 at 9:34 PM, Marvin Humphrey <marvin@rectangular.com> wrote:
> Greets,
>
> My laptop has two cores, but the Lucy build process is single threaded and
> doesn't take advantage of the second processor.
>
> I hacked up the patch below for trunk/perl/buildlib/Lucy/Build.pm to try to
> speed things up.  It forks off a max of 4 child processes which compile up to
> 10 C files each.  Here are before-and-after results for "time ./Build code":

Sorry, I am a perl dummy, so forgive me if i interpreted your
statements/code wrong :)
why is it 10 C files each?
wouldn't this require some wasted fork() overhead respawning many children?
couldnt you instead just split all the .c files into 4 pieces, and
only have 4 children up front?

p.s. we do a similar thing in Lucene-java with running tests, and a
problem can be balancing the workload with the children.
one heuristic about how long a test will take to run, or file will
take to compile, is its length in bytes.
it might be useful to sort the list of files by their size in bytes,
and use mod to divide them up.

Mime
View raw message