couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nolan Lawson <no...@nolanlawson.com>
Subject Re: Getting libraries to test RCs
Date Fri, 02 Sep 2016 17:43:02 GMT
Thanks, Dale. That was my recollection as well.

Basically PouchDB does PUT -> DELETE -> PUT between every test, so since
there are 1000s of tests, this race condition comes up pretty easily. We
can add a timeout or do a random DB name, but without doing that we don't
know if Couch 2.x is truly "passing" the test suite or not.

I have some time this weekend, so I'll look into adding a patch to do the
workaround for Couch 2. I tend to side with Jan that in a clustered system
it can't reliably tell us when a database was truly deleted without
sacrificing the A in CAP. PouchDB users are already familiar with the weird
ways that databases start to behave when you actually DELETE them (e.g.
replication gets unreliable), hence workarounds like
https://www.npmjs.com/package/pouchdb-erase . In practice I expect PouchDB
users to never delete databases, so this is just an artifact of our test
suite IMO.

–Nolan


On Fri, Sep 2, 2016 at 3:14 AM, Dale Harvey <dale@arandomurl.com> wrote:

> In PouchDB we can look into a workaround that uses random names only when
> the tests are run against Couch 2.0, however I would really like to make
> sure that a database not being fully deleted when we get a successful
> confirmation of deletion is considered a bug, it has impacts beyond the
> test suite, its really hard to create a reliable system when there is no
> way for you to be certain when a database is deleted.
>
> Will found it easiest to reproduce this using concurrent scripts but would
> like to clarify that Pouch doesnt run the test suite in parallel, this bug
> can be hit by doing CREATE -> DELETE -> CREATE, its extremely hard to nail
> down and reproduce (the similiar bug in PouchDB took many attempts +
> months). I will take a look at seeing if I can make an easier and clearer
> steps to reproduce.
>
> On 2 September 2016 at 11:01, Jan Lehnardt <jan@apache.org> wrote:
>
> >
> > > On 02 Sep 2016, at 11:58, Will Holley <willholley@gmail.com> wrote:
> > >
> > > Jan - I can understand that being the case in a clustered setup with
> > > distributed shard maps but shouldn't n=1 mitigate that?
> >
> > n=1 still does q=8 (8 shards per node) and the software makes
> > noconsistency guarantees whatsoever.
> >
> > n=1 && q=1 might work as a side-effect, but not sure how that is useful
> > for reliable tests :)
> >
> > Best
> > Jan
> > --
> >
> >
> > >
> > > On 2 September 2016 at 10:53, Jan Lehnardt <jan@apache.org> wrote:
> > >>
> > >>> On 02 Sep 2016, at 11:45, Dale Harvey <dale@arandomurl.com> wrote:
> > >>>
> > >>> In PouchDB we used to generate unique database names for tests,
> > however we
> > >>> removed it for serveral reasons, one large reason being it indicates
> a
> > race
> > >>> condition in critical code if we cannot reliably create -> delete
->
> > create
> > >>> the same database (we have uncovered and fixed a lot of bugs in
> > PouchDB due
> > >>> to this). While its not my call how to prioritise those bugs, I
> really
> > do
> > >>> not think we should be closing what are fairly serious bugs because
> it
> > >>> wasnt inconvenient to workaround them in the couch test suite.
> > >>
> > >> It’s just that a CouchDB 2.0 cluster is an AP system, and recreating
> > databases
> > >> in quick succession reliably basically requires a CA system and that’s
> > not what can do easily.
> > >>
> > >> (I hope I got the CAP letters right, but I think it is clear what I
> > mean)
> > >>
> > >> That is, maybe we skip those tests when run against a CouchDB 2.0
> > endpoint and keep them for PouchDB?
> > >>
> > >> Best
> > >> Jan
> > >> --
> > >>
> > >>
> > >>>
> > >>> On 2 September 2016 at 10:31, Joan Touzet <wohali@apache.org>
wrote:
> > >>>
> > >>>> Hi Nolan, Will:
> > >>>>
> > >>>> A further update from looking deeper with @janl. It appears that
we
> > >>>> have a pending fix for COUCHDB-3017 and we'll work on getting that
> > >>>> merged before 2.0.
> > >>>>
> > >>>> COUCHDB-3034 is a WONTFIX. FYI in CouchDB itself we changed all
of
> > >>>> our tests to use unique database names. I'll update the bug myself
> > >>>> shortly.
> > >>>>
> > >>>> -Joan
> > >>>>
> > >>>> ----- Original Message -----
> > >>>>> From: "Joan Touzet" <wohali@apache.org>
> > >>>>> To: dev@couchdb.apache.org
> > >>>>> Sent: Friday, September 2, 2016 5:15:00 AM
> > >>>>> Subject: Re: Getting libraries to test RCs
> > >>>>>
> > >>>>> Hi Will,
> > >>>>>
> > >>>>> Neither of these are currently tagged as blocking issues for
> CouchDB
> > >>>>> 2.0, only major priority. If you want to flag them as such,
this is
> > >>>>> your last chance, and even still, there's no guarantee fixes
for
> them
> > >>>>> will hit 2.0.
> > >>>>>
> > >>>>> Erlangers, is there any chance of at least triaging these today?
> > >>>>>
> > >>>>> -Joan
> > >>>>>
> > >>>>> ----- Original Message -----
> > >>>>>> From: "Will Holley" <willholley@gmail.com>
> > >>>>>> To: dev@couchdb.apache.org, "Joan Touzet" <wohali@apache.org>
> > >>>>>> Sent: Friday, September 2, 2016 4:43:48 AM
> > >>>>>> Subject: Re: Getting libraries to test RCs
> > >>>>>>
> > >>>>>> Assuming nothing's changed in the last few weeks, there
are 2
> > >>>>>> issues
> > >>>>>> which cause the PouchDB tests to fail against master: COUCHDB-3017
> > >>>>>> and
> > >>>>>> COUCHDB-3034.
> > >>>>>>
> > >>>>>> Both could be addressed in the test suite by using different
> > >>>>>> database
> > >>>>>> names for each test, but that's quite a disruptive change.
> > >>>>>>
> > >>>>>> On 2 September 2016 at 03:15, Joan Touzet <wohali@apache.org>
> > >>>>>> wrote:
> > >>>>>>> Hi Nolan, you state that it's 'failing for known reasons.'
Is
> > >>>>>>> that
> > >>>>>>> reasons in PouchDB or anything you need to push back
on us? We'd
> > >>>>>>> like
> > >>>>>>> to know ASAP as we're very, very close to releasing
2.0 now.
> > >>>>>>>
> > >>>>>>> I have zero PouchDB knowledge so I'm hoping you can
give us a
> > >>>>>>> short
> > >>>>>>> summary of what you think is wrong.
> > >>>>>>>
> > >>>>>>> All the best,
> > >>>>>>> Joan
> > >>>>>>>
> > >>>>>>> ----- Original Message -----
> > >>>>>>>> From: "Nolan Lawson" <nolan@nolanlawson.com>
> > >>>>>>>> To: dev@couchdb.apache.org
> > >>>>>>>> Sent: Thursday, September 1, 2016 7:56:42 PM
> > >>>>>>>> Subject: Re: Getting libraries to test RCs
> > >>>>>>>>
> > >>>>>>>> We have been testing CouchDB master in PouchDB
for months now,
> > >>>>>>>> but
> > >>>>>>>> as
> > >>>>>>>> an allowed failure because I believe it’s failing
for known
> > >>>>>>>> reasons.
> > >>>>>>>> We test both using Node.js and the browser.
> > >>>>>>>>
> > >>>>>>>> Node: https://travis-ci.org/pouchdb/pouchdb/jobs/156198210
> > >>>>>>>> Browser: https://travis-ci.org/pouchdb/pouchdb/jobs/156198211
> > >>>>>>>>
> > >>>>>>>> For anyone who wants to run the Pouch test suite
against
> > >>>>>>>> CouchDB,
> > >>>>>>>> it’s just:
> > >>>>>>>>
> > >>>>>>>> git clone https://github.com/pouchdb/pouchdb.git
> > >>>>>>>> cd pouchdb
> > >>>>>>>> npm I
> > >>>>>>>> COUCH_HOST=http://localhost:5984 BAIL=0 npm t
> > >>>>>>>>
> > >>>>>>>> BAIL=0 will tell it to run the full test suite
and not stop on
> > >>>>>>>> any
> > >>>>>>>> failures. That way you can inspect the failures
and see if
> > >>>>>>>> they’re
> > >>>>>>>> serious or not.
> > >>>>>>>>
> > >>>>>>>> Cheers,
> > >>>>>>>> Nolan
> > >>>>>>>>
> > >>>>>>>>> On Aug 29, 2016, at 12:15 PM, Jan Lehnardt
<jan@apache.org>
> > >>>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Anyone on this list who could help with this?
The work items
> > >>>>>>>>> are
> > >>>>>>>>> fairly self-explanatory and not very big individually
<3
> > >>>>>>>>>
> > >>>>>>>>> Best
> > >>>>>>>>> Jan
> > >>>>>>>>> --
> > >>>>>>>>>
> > >>>>>>>>>> On 10 Aug 2016, at 09:37, Jan Lehnardt
<jan@apache.org>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>> Hey everyone,
> > >>>>>>>>>>
> > >>>>>>>>>> from Joan’s excellent blog post about
testing Release
> > >>>>>>>>>> Candidates:
> > >>>>>>>>>>
> > >>>>>>>>>>> To our valued CouchDB application and
library developers:
> > >>>>>>>>>>> please,
> > >>>>>>>>>>> please run your software against each
of the options below.
> > >>>>>>>>>>
> > >>>>>>>>>> — https://blog.couchdb.org/2016/08/08/release-candidates/
> > >>>>>>>>>>
> > >>>>>>>>>> I think we can be a little more proactive
about this for
> > >>>>>>>>>> CouchDB
> > >>>>>>>>>> client libraries: let’s open issues on
all the
> > >>>>>>>>>> CouchDB-compatible
> > >>>>>>>>>> client software we care about to test an
RC.
> > >>>>>>>>>>
> > >>>>>>>>>> Since there are a lot of projects, and
we don’t necessarily
> > >>>>>>>>>> know
> > >>>>>>>>>> which one we “care” about, we should
try to be clever about
> > >>>>>>>>>> it.
> > >>>>>>>>>>
> > >>>>>>>>>> Maybe something like this can work:
> > >>>>>>>>>>
> > >>>>>>>>>> 1. We prepare an issue text explaining
the thing: Heya,
> > >>>>>>>>>> CouchDB
> > >>>>>>>>>> team here, major new version coming up,
you should test it
> > >>>>>>>>>> like
> > >>>>>>>>>> so: <include instructions to test against
a 3-node cluster.
> > >>>>>>>>>> Maybe
> > >>>>>>>>>> even provide a cluster to do this, or Cloudant
can sponsor
> > >>>>>>>>>> something?
> > >>>>>>>>>>
> > >>>>>>>>>> 2. Post this message with a call to action
on user@c.a.o, the
> > >>>>>>>>>> weekly news, and our other (social) media
channels.
> > >>>>>>>>>>
> > >>>>>>>>>> 3. Ask people who submitted an issue to
report back with a
> > >>>>>>>>>> link.
> > >>>>>>>>>>
> > >>>>>>>>>> 4. Collect the link in an issue or JIRA
(this could be done
> > >>>>>>>>>> in
> > >>>>>>>>>> 3.,
> > >>>>>>>>>> but then everybody needs to be added to
the wiki write group,
> > >>>>>>>>>> and
> > >>>>>>>>>> that’s just extra overhead we don’t
need). Maybe we borrow a
> > >>>>>>>>>> gist
> > >>>>>>>>>> for this, or a Google doc.
> > >>>>>>>>>>
> > >>>>>>>>>> That way we encourage client software to
check out RCs and we
> > >>>>>>>>>> can
> > >>>>>>>>>> keep track, while the community helps to
select which
> > >>>>>>>>>> software
> > >>>>>>>>>> to
> > >>>>>>>>>> encourage to test 2.0 compat, and helps
spread the word and
> > >>>>>>>>>> the
> > >>>>>>>>>> burden is not left with just a few folks.
> > >>>>>>>>>>
> > >>>>>>>>>> What do you think?
> > >>>>>>>>>>
> > >>>>>>>>>> Best
> > >>>>>>>>>> Jan
> > >>>>>>>>>> --
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> --
> > >>>>>>>>> Professional Support for Apache CouchDB:
> > >>>>>>>>> https://neighbourhood.ie/couchdb-support/
> > >>>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>
> > >>>>>
> > >>>>
> > >>
> > >> --
> > >> Professional Support for Apache CouchDB:
> > >> https://neighbourhood.ie/couchdb-support/
> > >>
> >
> > --
> > Professional Support for Apache CouchDB:
> > https://neighbourhood.ie/couchdb-support/
> >
> >
>



-- 
Nolan Lawson
nolanlawson.com
github.com/nolanlawson

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message