hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Wang <andrew.w...@cloudera.com>
Subject Re: Local repo sharing for maven builds
Date Fri, 18 Sep 2015 22:14:13 GMT
Okay, some browsing of Jenkins docs [1] says that we could key the
maven.repo.local off of $EXECUTOR_NUMBER to do per-executor repos like
Bernd recommended, but that still requires some hook into test-patch.sh.

Regarding install, I thought all we needed to install was
hadoop-maven-plugins, but we do more than that now in test-patch.sh. Not
sure if we can reduce that.

[1]
https://wiki.jenkins-ci.org/display/JENKINS/Building+a+software+project#Buildingasoftwareproject-JenkinsSetEnvironmentVariables

On Fri, Sep 18, 2015 at 2:42 PM, Allen Wittenauer <aw@altiscale.com> wrote:

>
> The collisions have been happening for about a year now.   The frequency
> is increasing, but not enough to be particularly worrisome. (So I'm
> slightly amused that one blowing up is suddenly a major freakout.)
>
> Making changes to the configuration without knowing what one is doing is
> probably a bad idea. For example, if people are removing the shared cache,
> I hope they're also prepared for the bitching that is going to go with the
> extremely significant slow down caused by downloading the java prereqs for
> building for every test...
>
> As far as Yetus goes, we've got a JIRA open to provide for per-instance
> caches when using the docker container code. I've got it in my head how I
> think we can do it, but just haven't had a chance to code it.  So once that
> gets written up + turning on containers should make the problem go away
> without any significant impact on test time.  Of course, that won't help
> the scheduled builds but those happen at an even smaller rate.
>
>
> On Sep 18, 2015, at 12:19 PM, Andrew Wang <andrew.wang@cloudera.com>
> wrote:
>
> > Sangjin, you should have access to the precommit jobs if you log in with
> > your Apache credentials, even as a branch committer.
> >
> > https://builds.apache.org/job/PreCommit-HDFS-Build/configure
> >
> > The actual maven invocation is managed by test-patch.sh though.
> > test-patch.sh has a MAVEN_ARGS which looks like what we want, but I don't
> > think we can just set it before calling test-patch, since it'd get
> squashed
> > by setup_defaults.
> >
> > Allen/Chris/Yetus folks, any guidance here?
> >
> > Thanks,
> > Andrew
> >
> > On Fri, Sep 18, 2015 at 11:55 AM, <ecki@zusammenkunft.net> wrote:
> >
> >> You can use one per build processor, that reduces concurrent updates but
> >> still keeps the cache function. And then try to avoid using install.
> >>
> >> --
> >> http://bernd.eckenfels.net
> >>
> >> -----Original Message-----
> >> From: Andrew Wang <andrew.wang@cloudera.com>
> >> To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>
> >> Cc: Andrew Bayer <andrew.bayer@gmail.com>, Sangjin Lee <
> sjlee@twitter.com>,
> >> Lei Xu <lei@cloudera.com>, infrastructure@apache.org
> >> Sent: Fr., 18 Sep. 2015 20:42
> >> Subject: Re: Local repo sharing for maven builds
> >>
> >> I think each job should use a maven.repo.local within its workspace like
> >> abayer said. This means lots of downloading, but it's isolated.
> >>
> >> If we care about download time, we could also bootstrap with a tarred
> >> .m2/repository after we've run a `mvn compile`, so before it installs
> the
> >> hadoop artifacts.
> >>
> >> On Fri, Sep 18, 2015 at 11:02 AM, Ming Ma <mingma@twitter.com.invalid>
> >> wrote:
> >>
> >>> +hadoop common dev. Any suggestions?
> >>>
> >>>
> >>> On Fri, Sep 18, 2015 at 10:41 AM, Andrew Bayer <andrew.bayer@gmail.com
> >
> >>> wrote:
> >>>
> >>>> You can change your maven call to use a different repository - I
> >> believe
> >>>> you do that with -Dmaven.repository.local=path/to/repo
> >>>> On Sep 18, 2015 19:39, "Ming Ma" <mingma@twitter.com> wrote:
> >>>>
> >>>>> Hi,
> >>>>>
> >>>>> We are seeing some strange behaviors in HDFS precommit build. It
> seems
> >>>>> like it is caused by the local repo on the same machine being used
by
> >>>>> different concurrent jobs which can cause issues.
> >>>>>
> >>>>> In HDFS, the build and test of "hadoop-hdfs-project/hdfs" depend
on
> >>>>> "hadoop-hdfs-project/hdfs-client"'s  hadoop-hdfs-client-3.0.0-
> >>>>> SNAPSHOT.jar. HDFS-9004 adds some new method to
> >>> hadoop-hdfs-client-3.0.0-SNAPSHOT.jar.
> >>>>> In the precommit build for HDFS-9004, unit tests for
> >>> "hadoop-hdfs-project/hdfs"
> >>>>> complain the method isn't defined
> >>>>> https://builds.apache.org/job/PreCommit-HDFS-Build/12522/testReport/
> .
> >>>>> Interestingly sometimes it just works fine
> >>>>> https://builds.apache.org/job/PreCommit-HDFS-Build/12507/testReport/
> .
> >>>>>
> >>>>> So we are suspecting that there is another job running at the same
> >> time
> >>>>> that published different version of
> >>> hadoop-hdfs-client-3.0.0-SNAPSHOT.jar
> >>>>> which doesn't have the new methods defined to the local repo which
is
> >>>>> shared by all jobs on that machine.
> >>>>>
> >>>>> If the above analysis is correct, what is the best way to fix the
> >> issue
> >>>>> so that different jobs can use their own maven local repo for build
> >> and
> >>>>> test?
> >>>>>
> >>>>> Thanks.
> >>>>>
> >>>>> Ming
> >>>>>
> >>>>
> >>>
> >>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message