avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Niels Basjes (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-1537) Make it easier to set up a multi-language build environment
Date Thu, 18 Dec 2014 11:22:13 GMT

    [ https://issues.apache.org/jira/browse/AVRO-1537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14251528#comment-14251528

Niels Basjes commented on AVRO-1537:

Hi [~tomwhite], 
I verified on my system but running the scipt for the first time takes 'ages' and running
it the second time I see the command prompt within two seconds.
So if you get very different results I suspect a problem on your system.

I agree that the 'safetest' name is a poor choice. Perhaps simply having the {{./build.sh
shell}} is good enough for the first version?

I also noticed that all files in the avro directory become owned by {{root}} because that
is the default user inside docker.
So after exiting the docker environment I could not longer do a {{mvn clean}} because the
generated files (.class, .jar, etc) are all stored in directories owned by the root user.

In an attempt to solve this I came up with this.
I effectively create an additional layer that simply recreates the current user (same username,
userid, groupid) in the docker environment.

function runindocker {
    docker build -t ${DOCKER_IMAGE_NAME} ${SOURCEDIR}/docker
    docker build -t ${DOCKER_IMAGE_NAME}_${USER} - << UserSpecificDocker
    RUN groupadd -g $(id -g) ${USER}
    RUN useradd  -g $(id -g) -u $(id -u) -k /root -m ${USER}
    ENV HOME /home/${USER}
    # By mapping the .m2 directory you can do an mvn install from within the container and
use the result on your normal system.
    # And this also is a significant speedup in subsequent builds because the dependencies
are downloaded only once.
    docker run --rm=true -t -i -v ${SOURCEDIR}:/home/${USER}/avro -w /home/${USER}/avro -v
${HOME}/.m2:/home/${USER}/.m2 -u ${USER} ${DOCKER_IMAGE_NAME}_${USER} $1 $2 $3 $4 $5 $6

> Make it easier to set up a multi-language build environment
> -----------------------------------------------------------
>                 Key: AVRO-1537
>                 URL: https://issues.apache.org/jira/browse/AVRO-1537
>             Project: Avro
>          Issue Type: Improvement
>            Reporter: Martin Kleppmann
>            Assignee: Tom White
>         Attachments: AVRO-1537.patch, AVRO-1537.patch, AVRO-1537.patch
> It's currently quite tedious to set up an environment in which the Avro test suites for
all supported languages can be run, and in which release candidates can be built. This is
especially so when we need to test against several different versions of a programming language
or VM (e.g. JDK6/JDK7/JDK8, Ruby 1.8.7/1.9.3/2.0/2.1).
> Our shared Hudson server isn't an ideal solution, because it only runs tests on changes
that are already committed, and maintenance of the server can't easily be shared across the
> I think a Docker image might be a good solution, since it could be set up by one person,
shared with all Avro developers, and maintained by the community on an ongoing basis. But
other VM solutions (Vagrant, for example?) might work just as well. Suggestions welcome.
> Related resources:
> * Using AWS (setting up an EC2 instance for Avro build and release): https://cwiki.apache.org/confluence/display/AVRO/How+To+Release#HowToRelease-UsingAWSforAvroBuildandRelease
> * Testing multiple versions of Ruby in CI: AVRO-1515

This message was sent by Atlassian JIRA

View raw message