drill-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Dunning <ted.dunn...@gmail.com>
Subject Re: Regarding unit test failures -- patterns & possible fixes
Date Tue, 07 Apr 2015 01:07:47 GMT
My mac knows it has 2 physical and 4 logical cores:

ted:spy$ sysctl hw | grep cpu
hw.ncpu: 4
hw.activecpu: 4
hw.physicalcpu: 2
hw.physicalcpu_max: 2
hw.logicalcpu: 4
hw.logicalcpu_max: 4
hw.cputype: 7
hw.cpusubtype: 4
hw.cpu64bit_capable: 1
hw.cpufamily: 526772277
hw.cpufrequency: 2900000000
hw.cpufrequency_min: 2900000000
hw.cpufrequency_max: 2900000000
hw.cputhreadtype: 1
hw.ncpu = 4
hw.cpufrequency = 2900000000
hw.availcpu = 4


On Mon, Apr 6, 2015 at 5:52 PM, Hanifi Gunes <hgunes@maprtech.com> wrote:

> @Ted, I actually have a patch pending review for this at
> https://issues.apache.org/jira/browse/DRILL-2039. However, @Hakeem you may
> still end up in a troubling situation as your mac might have 4 logical
> cores =)
>
> *sysctl hw | grep **cpu* should tell you.
>
>
> -Hanifi
>
>
> On Mon, Apr 6, 2015 at 5:22 PM, Ted Dunning <ted.dunning@gmail.com> wrote:
>
> > Maven allows the forkCount to be set in terms of the number of cores on
> the
> > machine by appending a C to the number.
> >
> > (I think)
> >
> >
> >
> > On Mon, Apr 6, 2015 at 5:15 PM, Abdel Hakim Deneche <
> adeneche@maprtech.com
> > >
> > wrote:
> >
> > > From my personal experience, using a forkCount of 4 makes some tests to
> > > timeout, not because there are some concurrent problems but just
> because
> > my
> > > machine is not powerful enough to run as many tests in parallel.
> > >
> > > On Mon, Apr 6, 2015 at 5:06 PM, Hanifi Gunes <hgunes@maprtech.com>
> > wrote:
> > >
> > > > Hey devs,
> > > >
> > > >
> > > > I have been dealing with sporadic unit test failures since last week.
> > I'd
> > > > like to share my findings on failing unit tests outlining the
> patterns
> > > > causing these.
> > > >
> > > > The general advice I was given was to run tests lowering the fork
> count
> > > > (the default is 4). This sounds ok to get around the problem. If this
> > is
> > > > your intent feel free to give a try. However, I wanted to investigate
> > > this
> > > > further, went ahead and set fork count to 8, number of cores testing
> > > > machine has.
> > > >
> > > > The first set of failures manifested around TestUnionAll &
> > > > TestExampleQueries. After Hakeem's pointer, I noticed that some test
> > > cases
> > > > in TestUnionAll & TestExampleQueries creates & drops views with
the
> > same
> > > > name. In a concurrent settings(given fork count > 0) with no strict
> > > > ordering among test cases, this sure creates a havoc throwing
> arbitrary
> > > > errors at each test run.
> > > https://issues.apache.org/jira/browse/DRILL-2684
> > > > takes care of this. After applying this fix, now I can cleanly run
> > > > java-exec tests, faster & consistently.
> > > >
> > > > Then my hive tests started failing since multiple test classes use
> the
> > > same
> > > > metadata folders and clean up after without caring whether other hive
> > > tests
> > > > are still running. I got a patch to unique-ify metadata folders for
> > test
> > > > class. https://issues.apache.org/jira/browse/DRILL-2685 tracks this
> > > > however
> > > > the patch is not public yet.
> > > >
> > > > The last problem relates to hive related tests living under jdbc
> > module.
> > > I
> > > > know that there has been an ongoing effort to migrate these tests
> from
> > > jdbc
> > > > module to where they belong. I am not sure about the timeframe for
> the
> > > > migration but I would think the sooner is better.
> > > >
> > > > To avoid further failures, I would strongly recommend devs to use
> view
> > > > names and external resources that are unique across test cases. One
> > idea
> > > is
> > > > to suffix view a view name with test case/class name for instance
> > prefer
> > > > using names_view_test_union_all rather than a more generic
> names_view.
> > > Also
> > > > with increasing number of test cases checked in, it takes more and
> more
> > > > time to complete a test run. We should consider concurrent test runs
> > as a
> > > > legitimate everyday use case and design tests accordingly.
> > > >
> > > >
> > > > I would be interested in hearing other unit test failure patterns so
> as
> > > to
> > > > be alert. Feel free to share if you discovered any.
> > > >
> > > >
> > > > Regards.
> > > > -Hanifi
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Abdelhakim Deneche
> > >
> > > Software Engineer
> > >
> > >   <http://www.mapr.com/>
> > >
> > >
> > > Now Available - Free Hadoop On-Demand Training
> > > <
> > >
> >
> http://www.mapr.com/training?utm_source=Email&utm_medium=Signature&utm_campaign=Free%20available
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message