hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Number of unit tests
Date Sun, 12 Oct 2014 23:15:23 GMT
The way this is supposed to work is that after applying the patch we run
mvn test on a dummy test to download all deps required for tests to the
maven directory. Then we rsync that maven repo directory to all the slaves.
This allows the mvn test commands to run in offline mode, which is faster.

We do this here:

https://github.com/apache/hive/blob/trunk/testutils/ptest2/src/main/resources/source-prep.vm#L85

This has worked for a long time so it's a mystery to me while that dep
would not be available. I just looked at the run on HIVE-8394 which failed
an hour ago (~3PM PST). The pom it's looking for has been there since noon.
I am rsyncing the maven directory to give this a try locally.

Brock

[hiveptest@ip-10-113-154-189 ~]$ date

Sun Oct 12 19:12:37 EDT 2014

[hiveptest@ip-10-113-154-189 ~]$ find
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/
-ls

2230753    4 drwxr-xr-x   5 hiveptest hiveptest     4096 Oct 12 12:07
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/

2230754    4 drwxr-xr-x   2 hiveptest hiveptest     4096 Oct 12 12:04
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/3

2231658    4 -rw-r--r--   1 hiveptest hiveptest      172 Oct 12 12:04
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/3/_maven.repositories

2231659    4 -rw-r--r--   1 hiveptest hiveptest     3351 Oct 12 12:04
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/3/oss-parent-3.pom

2231660    4 -rw-r--r--   1 hiveptest hiveptest       40 Oct 12 12:04
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/3/oss-parent-3.pom.sha1

2230756    4 drwxr-xr-x   2 hiveptest hiveptest     4096 Oct 12 19:11
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/7

2231666    8 -rw-r--r--   1 hiveptest hiveptest     4824 Oct 12 12:03
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/7/oss-parent-7.pom

2231667    4 -rw-r--r--   1 hiveptest hiveptest       40 Oct 12 12:03
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/7/oss-parent-7.pom.sha1

2231665    4 -rw-r--r--   1 hiveptest hiveptest      172 Oct 12 12:03
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/7/_maven.repositories

2230755    4 drwxr-xr-x   2 hiveptest hiveptest     4096 Oct 12 12:07
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/5

2231663    4 -rw-r--r--   1 hiveptest hiveptest       40 Oct 12 12:07
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/5/oss-parent-5.pom.sha1

2231661    4 -rw-r--r--   1 hiveptest hiveptest      172 Oct 12 12:07
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/5/_maven.repositories

2231662    4 -rw-r--r--   1 hiveptest hiveptest     4082 Oct 12 12:07
/home/hiveptest//54.198.245.146-hiveptest-1/maven/org/sonatype/oss/oss-parent/5/oss-parent-5.pom

On Sat, Oct 11, 2014 at 1:53 PM, Szehon Ho <szehon@cloudera.com> wrote:

> I dug a little bit into the logs, and it seems some kind of build error so
> these tests listed above are not running:
>
> [ERROR] Plugin org.antlr:antlr3-maven-plugin:3.4 or one of its
> dependencies could not be resolved: Failed to read artifact descriptor
> for org.antlr:antlr3-maven-plugin:jar:3.4: The repository system is
> offline but the artifact org.sonatype.oss:oss-parent:pom:7 is not
> available in the local repository. -> [Help 1]
>
>
> See:
>
> http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-1212/failed/
>
> Wonder if somebody knows about this?
>
> Thanks
> Szehon
>
> On Sat, Oct 11, 2014 at 12:42 PM, Thejas Nair <thejas@hortonworks.com>
> wrote:
>
> > The tests are also taking longer to complete - around 2 hr 50 mins, it
> > used by around 1hr 30 mins earlier. I wonder if the fewer test being
> > reported and longer duration are related (some kind of timeout ?)
> >
> >
> > On Fri, Oct 10, 2014 at 1:05 PM, David Schorow <dschorow@hortonworks.com
> >
> > wrote:
> > > Build #1188 had 6765 tests, #1189 had 4354 tests.  The following
> packages
> > > were not run in 1189.  I don't know why.
> > >
> > > Package
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/
> > >
> > > Total
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/
> > >
> > > org.apache.hadoop.hive.metastore.txn
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.metastore.txn/
> > >
> > > 57
> > > org.apache.hadoop.hive.ql.exec
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.exec/
> > >
> > > 48
> > > org.apache.hadoop.hive.ql.exec.errors
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.exec.errors/
> > >
> > > 4
> > > org.apache.hadoop.hive.ql.exec.mapjoin
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.exec.mapjoin/
> > >
> > > 1
> > > org.apache.hadoop.hive.ql.exec.persistence
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.exec.persistence/
> > >
> > > 14
> > > org.apache.hadoop.hive.ql.exec.tez
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.exec.tez/
> > >
> > > 12
> > > org.apache.hadoop.hive.ql.exec.vector
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.exec.vector/
> > >
> > > 150
> > > org.apache.hadoop.hive.ql.exec.vector.expressions
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.exec.vector.expressions/
> > >
> > > 235
> > > org.apache.hadoop.hive.ql.exec.vector.expressions.gen
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.exec.vector.expressions.gen/
> > >
> > > 1000
> > > org.apache.hadoop.hive.ql.exec.vector.udf
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.exec.vector.udf/
> > >
> > > 3
> > > org.apache.hadoop.hive.ql.io
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.io/
> > >
> > > 40
> > > org.apache.hadoop.hive.ql.io.orc
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.io.orc/
> > >
> > > 267
> > > org.apache.hadoop.hive.ql.io.parquet
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.io.parquet/
> > >
> > > 17
> > > org.apache.hadoop.hive.ql.io.parquet.serde
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.io.parquet.serde/
> > >
> > > 23
> > > org.apache.hadoop.hive.ql.io.parquet.serde.primitive
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.io.parquet.serde.primitive/
> > >
> > > 12
> > > org.apache.hadoop.hive.ql.io.sarg
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.io.sarg/
> > >
> > > 16
> > > org.apache.hadoop.hive.ql.lockmgr
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.lockmgr/
> > >
> > > 17
> > > org.apache.hadoop.hive.ql.lockmgr.zookeeper
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.lockmgr.zookeeper/
> > >
> > > 4
> > > org.apache.hadoop.hive.ql.optimizer.listbucketingpruner
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.optimizer.listbucketingpruner/
> > >
> > > 12
> > > org.apache.hadoop.hive.ql.optimizer.physical
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.optimizer.physical/
> > >
> > > 4
> > > org.apache.hadoop.hive.ql.parse.authorization
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.parse.authorization/
> > >
> > > 25
> > > org.apache.hadoop.hive.ql.parse.authorization.plugin.sqlstd
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.parse.authorization.plugin.sqlstd/
> > >
> > > 1
> > > org.apache.hadoop.hive.ql.plan
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.plan/
> > >
> > > 18
> > > org.apache.hadoop.hive.ql.processors
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.processors/
> > >
> > > 3
> > > org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd/
> > >
> > > 4
> > > org.apache.hadoop.hive.ql.session
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.session/
> > >
> > > 10
> > > org.apache.hadoop.hive.ql.tool
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.tool/
> > >
> > > 5
> > > org.apache.hadoop.hive.ql.udaf
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.udaf/
> > >
> > > 49
> > > org.apache.hadoop.hive.ql.udf
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.udf/
> > >
> > > 33
> > > org.apache.hadoop.hive.ql.udf.generic
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.udf.generic/
> > >
> > > 138
> > > org.apache.hadoop.hive.ql.udf.xml
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.udf.xml/
> > >
> > > 5
> > > org.apache.hadoop.hive.ql.util
> > > <
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1188/testReport/org.apache.hadoop.hive.ql.util/
> > >
> > > 3
> > >
> > >
> > >
> > >
> > > On 10/10/14, 11:19 AM, Szehon Ho wrote:
> > >>
> > >> I see vector and some other tests have stopped running now.  Couldn't
> > find
> > >> any reason from a quick glance.
> > >>
> > >> Thanks
> > >> Szehon
> > >>
> > >> On Fri, Oct 10, 2014 at 10:30 AM, Gunther Hagleitner<
> > >> gunther.hagleitner@gmail.com>  wrote:
> > >>
> > >>> Does anyone know why our test count has dropped recently from 6k to
> 4k?
> > >>>
> > >>> This shows the trend:
> > >>>
> > >>>
> > >>>
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/
> > >>>
> > >>> And this is the first one with 4k tests:
> > >>>
> > >>>
> > >>>
> >
> http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/1189/testReport/
> > >>>
> > >>> I don't see any obvious difference in the test breakdown.
> > >>>
> > >>> Thanks,
> > >>> Gunther.
> > >>>
> > >>
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> > reader of
> > > this message is not the intended recipient, you are hereby notified
> that
> > any
> > > printing, copying, dissemination, distribution, disclosure or
> forwarding
> > of
> > > this communication is strictly prohibited. If you have received this
> > > communication in error, please contact the sender immediately and
> delete
> > it
> > > from your system. Thank You.
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message