accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dylan Hutchison <dhutc...@cs.washington.edu>
Subject Re: Dealing with FastBulkImportIT
Date Sat, 13 Aug 2016 20:31:54 GMT
Hi Josh,

Forgive me for the design question, but shouldn't we distinguish tests of
correctness from tests of performance? The following is my understanding of
test categories, which does not totally align with Accumulo's test suite:

* Unit tests test individual components.
* Integration tests test using components together. They may require more
resources such as starting an Accumulo (MAC or real).
* Examples are executable code separate from the above, that an outside
developer or user can read to see how Accumulo is used. Examples have their
own tests.
* Performance evaluations are executable code separate from the above. They
range in complexity from simple "test bulk imports" to RabdomWalk with
agitation.

If performance evaluations run separately, then developers can treat then
like benchmarks, comparing times to those on similar hardware or across
commits.

Could you remind me of the reasons why we keep performance tests in the
standard set of ITs?

On Aug 13, 2016 1:03 PM, "Josh Elser" <josh.elser@gmail.com> wrote:

> I had assumed this test would pass locally (early-2013 MBP, 2.7 GHz Intel
> Core i7, 16G ram), but nope! 38s and 45+ seconds on two runs.
>
> Josh Elser wrote:
>
>> Hi,
>>
>> I have some complaints about FastBulkImportIT (a test added with
>> https://issues.apache.org/jira/browse/ACCUMULO-3327) but no good ideas
>> for how to better test it. As it presently stands, it is a very
>> subjective test WRT the kind of hardware used to run it.
>>
>> The test launches a 3-tserver MAC instance, creates about 585 splits on
>> a table, creates 100 files with ~1200 key-value pairs, and then waits
>> for the table to be balanced.
>>
>> At this point, it imports these files into that table and fails if that
>> takes longer than 30s.
>>
>> On my VPS (3core, 6G ram, "SSD"), the bulk import takes ~45 seconds.
>> This test will never pass on this node which bothers me because I am of
>> the opinion that anyone (with reasonable hardware) should be able to run
>> our tests (and to make sure it's clear: I believe this is reasonable
>> hardware).
>>
>> Does anyone have any thoughts on how we could stabilize this test for
>> developers?
>>
>> - Josh
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message