couchdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Garren Smith <gar...@apache.org>
Subject Re: [PROPOSAL] Remove `make elixir` from `make check`
Date Sat, 14 Dec 2019 17:29:28 GMT
I’m a bit nervous to remove the full elixir test suite. The only partition
tests are in the elixir suite. So we would have no test coverage for any of
that if we remove this. Given that we want to make this the best CouchDB
release ever it feels counter-intuitive to them be removing tests.

I also think there is a few JS tests we have disabled because they are
covered in Elixir so we would need to re-enable those in the same PR that
we remove the elixir tests.

Cheers
Garren

On Sat, Dec 14, 2019 at 2:51 AM Jay Doane <jaydoane@apache.org> wrote:

> On Thu, Dec 12, 2019 at 11:27 PM Joan Touzet <wohali@apache.org> wrote:
>
> > Jay, consider this a +1 to remove those if you can get to them before I
> do.
> >
>
> Joan, I don't think I was clear, but when I said
> > I'm +1 to removing these tests temporarily.
> by "these tests" I actually meant all the elixir tests, not just the two I
> mentioned. I don't think we'd get very far by just disabling those two, as
> there are many others that fail intermittently as well. I'm +1 to remove
> them, and here's a PR in case someone wants it to happen sooner rather than
> later:
>
> https://github.com/apache/couchdb/pull/2362
>
> Thanks,
> Jay
>
> >
> > If this makes the Elixir tests useful again, then it's a good compromise.
> >
> > -Joan
> >
> > On 2019-12-13 2:18 a.m., Jay Doane wrote:
> > > I've been trying to merge into master for the last 24+ hours, and have
> > > restarted the build about 10 times so far, but it has yet to pass.
> > >
> > > I kept track of failures, and these came up numerous times:
> > >
> > >   1) test GET /dbname/_design_docs (PartitionDDocTest)
> > >       test/elixir/test/partition_ddoc_test.exs:160
> > >       ** (RuntimeError) timed out after 10294 ms
> > >       code: retry_until(
> > >       stacktrace:
> > >         (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
> > > Couch.DBTest.retry_until/4
> > >         test/elixir/test/partition_ddoc_test.exs:163: (test)
> > >
> > >   1) test All Docs tests (AllDocsTest)
> > >       test/elixir/test/all_docs_test.exs:15
> > >       ** (RuntimeError) timed out after 30097 ms
> > >       code: retry_until(fn ->
> > >       stacktrace:
> > >         (couchdbtest) test/elixir/lib/couch/db_test.ex:301:
> > > Couch.DBTest.retry_until/4
> > >         test/elixir/test/all_docs_test.exs:75: (test)
> > >
> > > which both seem to be timing out.
> > >
> > > I'm +1 to removing these tests temporarily.
> > >
> > > Jay
> > >
> > > On Thu, Dec 12, 2019 at 1:39 PM Joan Touzet <wohali@apache.org> wrote:
> > >
> > >> Hi again,
> > >>
> > >> As I've been looking more closely at the CI suite for the Jenkins
> > >> transition, I've noticed that our Elixir test cases are actually the
> > >> most likely to fail. In 6 consecutive CI runs, 5 runs failed due to
> > >> failures in the Elixir suite. (The 6th failed due to a JS test
> failure.)
> > >>
> > >> We started the Elixir effort to retire the JS suite. We reached a
> > >> decision some months ago to put it into `make check` so that people
> > >> would pay attention to its output, and work to fix those tests,
> > >> accelerating our chances to get rid of the JS suite.
> > >>
> > >> Unfortunately, that's not materialised. Our Elixir test porters seem
> to
> > >> have stopped their work for a while now, and no one is systematically
> > >> addressing the failures in that suite. I've also heard other
> developers
> > >> mention (via IRC) that some of the test cases hold invalid assumptions
> > >> about how CouchDB works, especially with the Erlang-based clustering
> > >> code. It sounds to me like the effort needs a full code review.
> > >>
> > >> With 3.0 around the corner, I want people to be able to trust the
> output
> > >> of `make check` when downloading the tarball. If there is no
> objection,
> > >> when I merge the Erlang version / CI changes on Monday, I will also
> > >> comment out the call to `make elixir` as part of make check.
> > >>
> > >> When the Elixir porting team is more confident in the reliability and
> > >> completeness of their work, and we can successfully retire the JS
> suite,
> > >> we can reconsider.
> > >>
> > >> -Joan "really wanting to see green, but only seeing red" Touzet
> > >>
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message