impala-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Lipcon <t...@cloudera.com>
Subject Re: status-benchmark.cc compilation time
Date Thu, 23 Feb 2017 18:23:09 GMT
One thing we've found useful in Kudu to prevent bitrot of benchmarks is to
actually use gtest and gflags for the benchmark programs.

We set some flag like --benchmark_num_rows or --benchmark_num_iterations
with a default that's low enough to only run for a second or two, and run
it as part of our normal test suite. Rarely catches any bugs, but serves to
make sure that the code keeps working. Then, when a developer wants to
actually test a change for performance, they can run it with
--num_iterations=<high number>.

Doesn't help the weird case of status-benchmark where *compiling* takes 10
minutes... but I think the manual unrolling of 1000 status calls in there
is probably unrealistic anyway regarding how the different options perform
in a whole-program setting.

-Todd

On Thu, Feb 23, 2017 at 10:20 AM, Zachary Amsden <zamsden@cloudera.com>
wrote:

> Yes.  If you take a look at the benchmark, you'll notice the JNI call to
> initialize the frontend doesn't even have the right signature anymore.
> That's one easy way to bitrot while still compiling.
>
> Even fixing that isn't enough to get it off the ground.
>
>  - Zach
>
> On Tue, Feb 21, 2017 at 11:44 AM, Henry Robinson <henry@cloudera.com>
> wrote:
>
> > Did you run . bin/set-classpath.sh before running expr-benchmark?
> >
> > On 21 February 2017 at 11:30, Zachary Amsden <zamsden@cloudera.com>
> wrote:
> >
> > > Unfortunately some of the benchmarks have actually bit-rotted.  For
> > > example, expr-benchmark compiles but immediately throws JNI exceptions.
> > >
> > > On Tue, Feb 21, 2017 at 10:55 AM, Marcel Kornacker <
> marcel@cloudera.com>
> > > wrote:
> > >
> > > > I'm also in favor of not compiling it on the standard commandline.
> > > >
> > > > However, I'm very much against allowing the benchmarks to bitrot. As
> > > > was pointed out, those benchmarks can be valuable tools during
> > > > development, and keeping them in working order shouldn't really
> impact
> > > > the development process.
> > > >
> > > > In other words, let's compile them as part of gvo.
> > > >
> > > > On Tue, Feb 21, 2017 at 10:50 AM, Alex Behm <alex.behm@cloudera.com>
> > > > wrote:
> > > > > +1 for not compiling the benchmarks in -notests
> > > > >
> > > > > On Mon, Feb 20, 2017 at 7:55 PM, Jim Apple <jbapple@cloudera.com>
> > > wrote:
> > > > >
> > > > >> > On which note, would anyone object if we disabled benchmark
> > > > compilation
> > > > >> by
> > > > >> > default when building the BE tests? I mean separating out
> -notests
> > > > into
> > > > >> > -notests and -build_benchmarks (the latter false by default).
> > > > >>
> > > > >> I think this is a great idea.
> > > > >>
> > > > >> > I don't mind if the benchmarks bitrot as a result, because
we
> > don't
> > > > run
> > > > >> > them regularly or pay attention to their output except when
> > > > developing a
> > > > >> > feature. Of course, maybe an 'exhaustive' run should build
the
> > > > benchmarks
> > > > >> > as well just to keep us honest, but I'd be happy if 95%
of
> Jenkins
> > > > builds
> > > > >> > didn't bother.
> > > > >>
> > > > >> The pre-merge (aka GVM aka GVO) testing builds
> > > > >> http://jenkins.impala.io:8080/job/all-build-options, which builds
> > > > >> without the "-notests" flag.
> > > > >>
> > > >
> > >
> >
> >
> >
> > --
> > Henry Robinson
> > Software Engineer
> > Cloudera
> > 415-994-6679
> >
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message