hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Allen Wittenauer ...@altiscale.com>
Subject Re: Local repo sharing for maven builds
Date Wed, 07 Oct 2015 15:39:29 GMT

yetus-5 was just committed which does all of this (and more, of course).

On Oct 6, 2015, at 2:35 AM, Steve Loughran <stevel@hortonworks.com> wrote:

> 
>> On 5 Oct 2015, at 19:45, Colin McCabe <cmccabe@alumni.cmu.edu> wrote:
>> 
>> On Mon, Sep 28, 2015 at 12:52 AM, Steve Loughran <stevel@hortonworks.com> wrote:
>>> 
>>> the jenkins machines are shared across multiple projects; cut the executors to
1/node and then everyone's performance drops, including the time to complete of all jenkins
patches, which is one of the goals.
>> 
>> Hi Steve,
>> 
>> Just to be clear, the proposal wasn't to cut the executors to 1 per
>> node, but to have multiple Docker containers per node (perhaps 3 or 4)
>> and run each executor in an isolated container.  At that point,
>> whatever badness Maven does on the .m2 stops being a problem for
>> concurrently running jobs.
>> 
> 
> I'd missed that bit. Yes, something with a containerized ~//m2 repo gets the isolation
without playing with mvn  version fixup
> 
>> I guess I don't feel that strongly about this, but the additional
>> complexity of the other solutions (like running a "find" command in
>> .m2, or changing artifactID) seems like a disadvantage compared to
>> just using multiple containers.  And there may be other race
>> conditions here that we're not aware of... like a TOCTOU between
>> checking for a jar in .m2 and downloading it, for example.  The
>> Dockerized solution skips all those potential failure modes and
>> complexity.
>> 
>> cheers,
>> Colin
>> 
> 


Mime
View raw message